Description of problem: Seeing this on a test koji system with a single builder. Both i386 and x86_64 builds are kicked off at the same time. One will usually fail at the repoquery step with: DEBUG util.py:314: Executing command: /usr/bin/repoquery -c /var/lib/mock/fedora-rawhide-i386/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/fedora-rawhide-i386/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} DEBUG util.py:264: Existing lock /var/run/yum.pid: another copy is running as pid 9305. DEBUG util.py:354: Child return code was: 50 Version-Release number of selected component (if applicable): mock-1.1.32-1.fc19.noarch How reproducible: Pretty often when two identical builds are started at the same time. Some thoughts: - Should repoquery be locking at all with /var/run/yum.pid? - If it does normally, perhaps it doesn't if -C is used, and perhaps it should? - Do we have to run it in the chroot?
Bringing in the yum-utils folks.
Yes, locking was added intentionally to fix BZ 969776. It's necessary since repoquery can update the metadata if it's too old. Possible solutions are: a) run "repoquery -C" as user, then it won't download/lock. Probably a "yum makecache" should be also run once, to make sure the cache is current. b) add a sleep-and-retry loop to repoquery locking code. Note that this might block for very long time, and some users may wish to abort instead, so maybe c) is better: c) handle the return code 50 in a wrapper and retry. Do you think b) should be implemented? Otherwise, it's not a bug.
For a), mock has just installed and updated the buildroot with yum so the cache should already be fresh. However, even with -C locking is still done and the process exits immediately. I could see locking still being needed so that the repodata isn't updated underneath repoquery. For b), I'm not sure what the standard behavior for yum these days is. It might be helpful here. However, more to the point I think, why isn't this an issue for all of the yum operations that mock does? Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache. Shouldn't repoquery be run in the same way? Can it?
> even with -C locking is still done and the process exits immediately. Not when uid != 0. The intent was that user processes may use root's metadata, without blocking the root (that always does proper locking), at the (small) cost of a possible race. Maybe we could change the locking code so when *root* runs -C, we don't lock either. > I'm not sure what the standard behavior for yum these days is. When Yum can't lock the cachedir it shows the current lock owner, sleeps 2 seconds, and retries. If it takes too long, users may abort with ctrl-c. > However, more to the point I think, why isn't this an issue for all of the yum operations that mock does? Probably due to the above. But Yum is (mostly) interactive tool, and most of yum-utils are not, so we should not pollute it's output, and just exit with an error. > Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache. Shouldn't repoquery be run in the same way? Can it? Repoquery can't use custom cachedir. it's mostly just an alternative to "yum search" with more options and output that's easier to parse.
(In reply to Zdeněk Pavlas from comment #4) > > Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache. Shouldn't repoquery be run in the same way? Can it? > > Repoquery can't use custom cachedir. it's mostly just an alternative to "yum > search" with more options and output that's easier to parse. I think we're going to need to be able to do this. It looks like the way mock does this with yum is to mount /var/cache/mock/<config>/yum_cache into the the install root as /var/cache/yum, then runs yum with --installroot set, which then appears to cause the cache dir to be based off of that. So, I think repoquery is going to need to either accept the --installroot option or some other way to specify the cache directory. mock folks, please chime in...
> Repoquery can't use custom cachedir I was wrong.. Right, there's no --installroot repoquery option, but you can make a custom yum.conf, set installroot there, and use repoquery --config option. Adding --installroot option would not be that hard, too.
--installroot would probably be the easiest for mock to support - since it already uses that and the yum.conf that is uses already refer to /var/cache/root (inside the install root).
*** Bug 988053 has been marked as a duplicate of this bug. ***
So, so I went for the easiest solution.. added a sleep-and-retry loop to repoquery locking code. Repoquery should poll the yum.pid lock instead of failing immediately.
While that is helpful, for mock we really need a way to specify what cache directory repoquery uses. As it stand the yum cache data for *every* different mock configuration (i.e. many different OSes and versions) gets dumped into the /var/cache/yum directory of the mock host. This is clearly wrong. Especially when the cache data is already available in /var/cache/mock/<chroot>/yum_cache.
Created attachment 780136 [details] Patch to use repoquery --installroot So, here's my attempt at making use of the installroot option and -C options. Seems to be working for me. I'm assuming that the old _installedPreBuildHook code was creating the tmp mock file to work around the lack on an --installroot option - I'm not sure if that worked or not. I don't know why the old _availablePostYumHook code didn't use the same trick. It seems worth it to me to use the -C option since we have just done an install and/or update.
Created attachment 780137 [details] Patch to use repoquery --installroot Sorry, messed that one up.
Hmm, with -C in the _availablePostYumHook I sometimes get messages like: DEBUG util.py:264: Cannot find repomd.xml file for rawhide/x86_64 Not sure why yum and repoquery appear to be out of sync here or who the repomd.xml isn't saved. But perhaps we can't use -C there.
Can't tell.. but the code that raises the "Cannot find repomd.xml" exception seems to be correct. With -C we should not download metadata, and there's definitely no repomd.xml in the cachedir, so we have to bail out. (the actual cachedir should be inside the installroot, but I might have missed something in http://lists.baseurl.org/pipermail/yum-devel/2013-July/010279.html if you can reproduce and strace this, please check)
Am I correct in presuming that this is going to end up as: 1) modify repoquery/yum with some new options for specifying chroot location 2) teach mock to use these new options
The primary problem was that repoquery uses metadata read/write, but it didn't lock at all. Then it did lock, but handled failure with immediate abort, and users expect it to retry. The --installroot option is (AIUI) mostly just UI stuff that simplifies some repoquery use cases.
I think the primary problem was that repoquery assumed you were always operating on the system metadata, and didn't have a way to operate on metadata in a different directory. Clark - That seem correct. Any progress here?
If I'm interpreting things correctly, repoquery is getting an --installroot option and once that happens I'll modify the package_state plugin for mock to use that and we'll all be happy. Or maybe just not as grumpy...
Already there: * Wed Aug 07 2013 Zdenek Pavlas <zpavlas> - 1.1.31-17 - Update to latest HEAD - Use new findRepos() API for yum-config-manager. BZ 971599 - repoquery: add --installroot option. BZ 988429
Created attachment 845508 [details] patch to make package state plugin use new repoquery --installroot option Ugh, I rolled this patch in before seeing that Orion had added a patch that does the same thing. Sorry, would have used it if I'd seen it :( Anyway, this passes all my unit-tests, so it's in for the 1.1.36 build, which should come out next week.
mock-1.1.36-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/mock-1.1.36-1.fc19
mock-1.1.36-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/mock-1.1.36-1.fc20
mock-1.1.36-1.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/mock-1.1.36-1.el6
Package mock-1.1.36-1.el6: * should fix your issue, * was pushed to the Fedora EPEL 6 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=epel-testing mock-1.1.36-1.el6' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-EPEL-2014-0467/mock-1.1.36-1.el6 then log in and leave karma (feedback).
mock-1.1.36-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.1.37-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/mock-1.1.37-1.fc19
mock-1.1.37-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/mock-1.1.37-1.fc20
mock-1.1.37-1.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/mock-1.1.37-1.el6
mock-1.1.37-2.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/mock-1.1.37-2.fc20
mock-1.1.37-2.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/mock-1.1.37-2.fc19
mock-1.1.37-2.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/mock-1.1.37-2.el6
mock-1.1.38-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/mock-1.1.38-1.fc19
mock-1.1.38-1.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/mock-1.1.38-1.el6
mock-1.1.38-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/mock-1.1.38-1.fc20
mock-1.1.38-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.1.38-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.1.38-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.