Description of problem: Fedora koji builders run Fedora 21. after updating koji to a version that uses mergerepo_c epel builds that use external repos started to fail to init the buildroot. the error appears to be due to mergerepo doing something wrong DEBUG util.py:388: Installed size: 306 M DEBUG util.py:388: Error downloading packages: DEBUG util.py:388: rpmdevtools-6.8-1.el5.noarch: failed to retrieve toplink/packages/rpmdevtools/6.8/1.el5/noarch/rpmdevtools-6.8-1.el5.noarch.rpm from build DEBUG util.py:388: error was [Errno 2] Local file does not exist: /var/tmp/koji/tasks/8528/9698528/repo_482068_premerge/toplink/packages/rpmdevtools/6.8/1.el5/noarch/rpmdevtools-6.8-1.el5.noarch.rpm DEBUG util.py:388: fakeroot-libs-1.12.2-21.el5.2.x86_64: failed to retrieve toplink/packages/fakeroot/1.12.2/21.el5.2/x86_64/fakeroot-libs-1.12.2-21.el5.2.x86_64.rpm from build DEBUG util.py:388: error was [Errno 2] Local file does not exist: /var/tmp/koji/tasks/8528/9698528/repo_482068_premerge/toplink/packages/fakeroot/1.12.2/21.el5.2/x86_64/fakeroot-libs-1.12.2-21.el5.2.x86_64.rpm DEBUG util.py:388: buildsys-macros-5-4.el5.noarch: failed to retrieve toplink/packages/buildsys-macros/5/4.el5/noarch/buildsys-macros-5-4.el5.noarch.rpm from build DEBUG util.py:388: error was [Errno 2] Local file does not exist: /var/tmp/koji/tasks/8528/9698528/repo_482068_premerge/toplink/packages/buildsys-macros/5/4.el5/noarch/buildsys-macros-5-4.el5.noarch.rpm DEBUG util.py:388: fakeroot-1.12.2-21.el5.2.x86_64: failed to retrieve toplink/packages/fakeroot/1.12.2/21.el5.2/x86_64/fakeroot-1.12.2-21.el5.2.x86_64.rpm from build DEBUG util.py:388: error was [Errno 2] Local file does not exist: /var/tmp/koji/tasks/8528/9698528/repo_482068_premerge/toplink/packages/fakeroot/1.12.2/21.el5.2/x86_64/fakeroot-1.12.2-21.el5.2.x86_64.rpm DEBUG util.py:499: Child return code was: 1 is the logs we see. the premerge repo apears to have the wrong baseurl in it. it could be that createrepo_c is wrong, but I suspect it is not or all builds would eb failing.
Hi Dennis, could you be more specific what the problem is (What you see in repodata and what do you expect)? Especially, could you provide me: * arguments that was used for mergerepo_c call. * the repodata generated by mergerepo_c and repodata generated by the previous tool so I could compare them? * the input repositories? * the expected baseurl (supposing the one in the output repodata is wrong)? * info if the input repositories was generated by createrepo_c? * info if the input repositories seem to be correct (the problem is most likely in mergerepo_c) or if also the intput repositories are wrong (the problem is most likely in the createrepo_c)
http://koji.fedoraproject.org/koji/taskinfo?taskID=9698520 is a koji task using createrepo_c and mergerepo_c The command to make the repo for the content in koji was /usr/bin/createrepo_c -vd -o /var/tmp/koji/tasks/8528/9698528/repo -i /mnt/koji/repos/dist-5E-epel-build/482068/x86_64/pkglist -g /mnt/koji/repos/dist-5E-epel-build/482068/groups/comps.xml --update --skip-stat /mnt/koji/repos/dist-5E-epel-build/482068/x86_64 the mergerepo_c command was /usr/bin/mergerepo_c --koji -a x86_64 -b /mnt/koji/repos/dist-5E-epel-build/482068/x86_64/blocklist -o /var/tmp/koji/tasks/8528/9698528/repo -g /mnt/koji/repos/dist-5E-epel-build/482068/groups/comps.xml -r file:///var/tmp/koji/tasks/8528/9698528/repo_482068_premerge/ -r http://kojipkgs.fedoraproject.org/repo/rhel/rhel-x86_64-server-5/ -r http://kojipkgs.fedoraproject.org/repo/rhel/rhel-x86_64-server-vt-5/ -r http://kojipkgs.fedoraproject.org/repo/rhel/rhel-x86_64-server-productivity-5/ there should be no references to /var/tmp/koji/tasks/8528/9698528/repo_482068_premerge in the repodata the data is not available via that location kojis mergerepo does the correct behaviour here. megrerepo_c should not be adding any baseurls, the expected behaviour as to what things should be is exactly what is done in koji's mergerepo. the repo with issues is teh one made by createrepo_c
I guess the --omit-baseurl option is what you are looking for. Could you please try to add the option to mergerepo_c call? If it provide expected result, I could modify the mergerepo_c to use the option implicitly when --koji option is pecified. Note: The option is available since v0.7.0.
I just had the same problem as Dennis with koji.spacewalkproject.org. (In reply to Tomas Mlcoch from comment #3) > I guess the --omit-baseurl option is what you are looking for. > Could you please try to add the option to mergerepo_c call? I just tried it, and it fixed the problem... but creates a new one. :-/ http://koji.spacewalkproject.org/kojifiles/work/tasks/6364/186364/root.log Taking one of these 404 packages, the newly generated repodata has this to say: <location href="Packages/n/ncurses-base-5.9-16.20140323.fc21.noarch.rpm"/> This is completely wrong, because the packages are **not** directly in the repositories, there is only a package list file, for example: http://koji.spacewalkproject.org/kojifiles/repos/spacewalk-nightly-fedora21-build/3198/x86_64/ Instead, using koji's mergerepos, this is what the generated repodata should look like for a package coming from an external repo: <location xml:base="http://dl.fedoraproject.org/pub/fedora/linux/releases/21/Everything/x86_64/os/Packages/n" href="ncurses-base-5.9-16.20140323.fc21.noarch.rpm"/> And this is wht it looks like for a package coming from an internal repo: <location xml:base="file:///var/tmp/koji/tasks/6273/186273/repo_3193_premerge/toplink/packages/rhnlib/2.5.76/1.fc21/noarch" href="rhnlib-2.5.76-1.fc21.noarch.rpm"/> This is what the --koji option needs to do.
<location href="toplink/packages/2048-cli/0.9/4.git20141214.723738c.el5/i386/2048-cli-0.9-4.git20141214.723738c.el5.i386.rpm"/> is what should be used for the internal koji repos <location xml:base="http://kojipkgs.fedoraproject.org/repo/rhel/rhel-i386-server-5/getPackage" href="zsh-html-4.2.6-9.el5.i386.rpm"/> is what should be in use for external repos kojis mergerepos has if reponum == 0 and not pkg.basepath: # this is the first repo (i.e. the koji repo) and appears # to be using relative urls #XXX - kind of a hack, but yum leaves us little choice #force the pkg object to report a relative location loc = """<location href="%s"/>\n""" % yum.misc.to_xml(pkg.remote_path, attrib=True) pkg._return_remote_location = make_const_func(loc) in its code, the koji mode needs to do the same thing, i.e. drop the base for the internal koji repo. which is always first.
Sorry, my previous comment was wrong for the location of packages from the internal repos. Dennis has it right, of course. :)
createrepo_c-0.8.2-1.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/createrepo_c-0.8.2-1.fc22
Mathieu, Dennis, could you please try v0.8.2?
hit a few issues. on the positive side an internal package <location href="toplink/packages/bash/4.3.33/3.fc22/x86_64/bash-4.3.33-3.fc22.x86_64.rpm"/> so that bit is right a package in an external repo however <location xml:base="file://http://kojipkgs.fedoraproject.org/repos/f22-build/latest/x86_64" href="toplink/packages/basesystem/10.0/10.fc21/noarch/basesystem-10.0-10.fc21.noarch.rpm"/> we should not go adding file:// infront of http:// that will cause issues $ /usr/bin/mergerepo_c --koji -a i386 -b /mnt/koji/repos/f23-build/533797/i386/blocklist -o /var/tmp/koji/tasks/7235/537235/repo -g /mnt/koji/repos/f23-build/533797/groups/comps.xml -r file:///var/tmp/koji/tasks/7235/537235/repo_533797_premerge/ -r http://kojipkgs.fedoraproject.org/repos/f23-build/latest/i386/ Warning: Cannot copy groupfile /mnt/koji/repos/f23-build/533797/groups/comps.xml on a rhel7 builder we hit the above warning which resulted in broken repos. we should exit with an abnormal exit signal, this will ensure koji fails the task
Package createrepo_c-0.8.2-1.fc22: * should fix your issue, * was pushed to the Fedora 22 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing createrepo_c-0.8.2-1.fc22' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-8228/createrepo_c-0.8.2-1.fc22 then log in and leave karma (feedback).
createrepo_c-0.8.3-1.fc22 that fixes the issue is ready: http://koji.fedoraproject.org/koji/taskinfo?taskID=9749033
Tomas, please add references to the bug reports in all your bodhi updates. As it is I thought you had built only for F22 (because there was no notice of an el6 build in this bug report), and I just wasted some time rebuilding for el6, only to find later that you had already built it and submitted an update. :( Going to try it out now, though.
So, for a package coming from an external repo, I now get: <location xml:base="http://dl.fedoraproject.org/pub/fedora/linux/updates/21/x86_64" href="0/0ad-0.0.18-1.fc21.x86_64.rpm"/> And for a package coming from an internal repo, I now get: <location href="toplink/packages/dnf-plugin-spacewalk/2.4.4/1.fc21/noarch/dnf-plugin-spacewalk-2.4.4-1.fc21.noarch.rpm"/> Which is all good. Everything ok for you as well Dennis?
Note that I'm not using the --omit-baseurl parameter any more, just --koji.
I spoke too fast. A new issue now is that the merged repo will contain the comps.xml file, however it isn't added to repomd.xml. As a result, builds fail because the 'build' and 'srpm-build' groups can't be found.
Any progress Tomas?
Fixed in upstream. The build will be ready tomorrow (22.5.)
Rawhide build is done: http://koji.fedoraproject.org/koji/taskinfo?taskID=9827626 Could you please try and let me know if it works? If so, I will do build for other fedora releases including epel ones.
Just rebuilt your new createrepo_c-0.8.3-2.fc23 for EL6 (as that's where I can test) and had a run with it: I'm running with Dennis' Koji patches to use createrepo_c/mergerepo_c, which means the mergerepo command ends up being: /usr/bin/mergerepo_c --koji -a x86_64 \ -b /mnt/koji/repos/spacewalk-nightly-fedora21-build/3220/x86_64/blocklist -o /var/tmp/koji/tasks/6949/186949/repo \ -g /mnt/koji/repos/spacewalk-nightly-fedora21-build/3220/groups/comps.xml \ -r file:///var/tmp/koji/tasks/6949/186949/repo_3220_premerge/ \ -r http://dl.fedoraproject.org/pub/fedora/linux/updates/21/x86_64/ \ -r ... For a package coming from an external repo, I now get: <location xml:base="http://dl.fedoraproject.org/pub/fedora/linux/updates/21/x86_64" href="0/0ad-0.0.18-1.fc21.x86_64.rpm"/> For a package coming from an internal repo, I get: <location href="toplink/packages/dnf-plugin-spacewalk/2.4.5/1.fc21/noarch/dnf-plugin-spacewalk-2.4.5-1.fc21.noarch.rpm"/> Both are correct. (as they were in comment 13) The comps file was properly added to the repodata: http://koji.spacewalkproject.org/kojifiles/repos/spacewalk-nightly-fedora21-build/3220/x86_64/repodata/ And it was also properly added to the repomd.xml file, which was the last issue I had. :) I've just done a scratch build from that repo and it passed the downloading steps just fine. So AFAICS mergerepo_c --koji is fixed with your latest build. Thanks!
Thanks for such extensive report! :) However, I've done one additional build, because two bugs were found (one in --all option and one in NVR merging method - the first one in --all also affects --koji option and may cause that some packages that should be included may be left out of the repodata). Rawhide build: http://koji.fedoraproject.org/koji/taskinfo?taskID=9838651 Epel 6 build: http://koji.fedoraproject.org/koji/taskinfo?taskID=9838668
As discussed with Tomas on IRC, I've been using the following build with success for the past few days: https://koji.fedoraproject.org/koji/buildinfo?buildID=639031
createrepo_c-0.9.0-1.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/createrepo_c-0.9.0-1.fc22
Createrepo_c 0.9.0 fully fixes this issue (and several more): Rawhide: http://koji.fedoraproject.org/koji/taskinfo?taskID=9862705 F22: http://koji.fedoraproject.org/koji/taskinfo?taskID=9862725 F21: http://koji.fedoraproject.org/koji/taskinfo?taskID=9862730 EPEL6: http://koji.fedoraproject.org/koji/taskinfo?taskID=9862737 EPEL7: http://koji.fedoraproject.org/koji/taskinfo?taskID=9862741
Hi Tomas, --koji should not use --all perhaps we should sit down and make sure that --koji is working as expected. I have not checked that all rpms for a srpm are included regardless of the repo they are in. additionally there needs to be no multilib with --koji
Hi Dennis, I agree with you. I would like to have a document that strictly defines how mergerepo_c with --koji option should behave - what is expected on input and how output should look like. The current implementation is based on code of mergerepos + some reverse engineering. It's because significant part of mergerepos behaviour is inherited from yum but yum's code is bloated and each version of yum behave slightly different. About the multilib are you sure? When I open koji/builder/mergerepos there is quite clearly evident that there is a MULTILIB_ARCHES dict which is used in architecture expansions and thus that multilib is honoured by mergerepos.
The comment here explains the expected behavior of mergerepos: https://git.fedorahosted.org/cgit/koji/tree/builder/mergerepos#n164 If you follow that it should be doing the right thing for Koji.
createrepo_c-0.9.0-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.