Cloning as the same repoquery code is in yum-utils-1.1.31-15.el7

+++ This bug was initially created as a clone of Bug #985681 +++

Description of problem:

Seeing this on a test koji system with a single builder.  Both i386 and x86_64 builds are kicked off at the same time.  One will usually fail at the repoquery step with:

DEBUG util.py:314:  Executing command: /usr/bin/repoquery -c /var/lib/mock/fedora-rawhide-i386/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/fedora-rawhide-i386/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
DEBUG util.py:264:  Existing lock /var/run/yum.pid: another copy is running as pid 9305.
DEBUG util.py:354:  Child return code was: 50

Version-Release number of selected component (if applicable):
mock-1.1.32-1.fc19.noarch

How reproducible:
Pretty often when two identical builds are started at the same time.

Some thoughts:
- Should repoquery be locking at all with /var/run/yum.pid?
- If it does normally, perhaps it doesn't if -C is used, and perhaps it should?
- Do we have to run it in the chroot?

--- Additional comment from Orion Poplawski on 2013-07-18 00:44:50 EDT ---

Bringing in the yum-utils folks.

--- Additional comment from Zdeněk Pavlas on 2013-07-18 04:06:12 EDT ---

Yes, locking was added intentionally to fix BZ 969776.  It's necessary since repoquery can update the metadata if it's too old.  Possible solutions are:

a) run "repoquery -C" as user, then it won't download/lock.  Probably a "yum makecache" should be also run once, to make sure the cache is current.

b) add a sleep-and-retry loop to repoquery locking code.

Note that this might block for very long time, and some users may wish to abort instead, so maybe c) is better:

c) handle the return code 50 in a wrapper and retry.

Do you think b) should be implemented?  Otherwise, it's not a bug.

--- Additional comment from Orion Poplawski on 2013-07-18 12:33:13 EDT ---

For a), mock has just installed and updated the buildroot with yum so the cache should already be fresh.  However, even with -C locking is still done and the process exits immediately.  I could see locking still being needed so that the repodata isn't updated underneath repoquery.

For b), I'm not sure what the standard behavior for yum these days is.  It might be helpful here.

However, more to the point I think, why isn't this an issue for all of the yum operations that mock does?  Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache.  Shouldn't repoquery be run in the same way?  Can it?

--- Additional comment from Zdeněk Pavlas on 2013-07-19 04:51:09 EDT ---

> even with -C locking is still done and the process exits immediately.

Not when uid != 0. The intent was that user processes may use root's metadata, without blocking the root (that always does proper locking), at the (small) cost of a possible race.  Maybe we could change the locking code so when *root* runs -C, we don't lock either.

> I'm not sure what the standard behavior for yum these days is.

When Yum can't lock the cachedir it shows the current lock owner, sleeps 2 seconds, and retries.  If it takes too long, users may abort with ctrl-c.

> However, more to the point I think, why isn't this an issue for all of the yum operations that mock does?

Probably due to the above.  But Yum is (mostly) interactive tool, and most of yum-utils are not, so we should not pollute it's output, and just exit with an error.

> Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache.  Shouldn't repoquery be run in the same way?  Can it?

Repoquery can't use custom cachedir. it's mostly just an alternative to "yum search" with more options and output that's easier to parse.

--- Additional comment from Orion Poplawski on 2013-07-19 11:50:08 EDT ---

(In reply to Zdeněk Pavlas from comment #4)
> > Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache.  Shouldn't repoquery be run in the same way?  Can it?
> 
> Repoquery can't use custom cachedir. it's mostly just an alternative to "yum
> search" with more options and output that's easier to parse.

I think we're going to need to be able to do this.  It looks like the way mock does this with yum is to mount /var/cache/mock/<config>/yum_cache into the the install root as /var/cache/yum, then runs yum with --installroot set, which then appears to cause the cache dir to be based off of that.

So, I think repoquery is going to need to either accept the --installroot option or some other way to specify the cache directory.

mock folks, please chime in...

--- Additional comment from Zdeněk Pavlas on 2013-07-22 04:45:25 EDT ---

> Repoquery can't use custom cachedir

I was wrong.. Right, there's no --installroot repoquery option, but you can make a custom yum.conf, set installroot there, and use repoquery --config option.  Adding --installroot option would not be that hard, too.

--- Additional comment from Orion Poplawski on 2013-07-22 11:20:35 EDT ---

--installroot would probably be the easiest for mock to support - since it already uses that and the yum.conf that is uses already refer to /var/cache/root (inside the install root).