Bug 985681 - repoquery writes /var/run/yum.pid, only one can run at a time
repoquery writes /var/run/yum.pid, only one can run at a time
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: mock (Show other bugs)
19
All Linux
unspecified Severity high
: ---
: ---
Assigned To: Clark Williams
Fedora Extras Quality Assurance
:
: 988053 (view as bug list)
Depends On: 988429
Blocks: 988223
  Show dependency treegraph
 
Reported: 2013-07-18 00:43 EDT by Orion Poplawski
Modified: 2014-04-19 05:19 EDT (History)
8 users (show)

See Also:
Fixed In Version: mock-1.1.38-1.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 988223 (view as bug list)
Environment:
Last Closed: 2014-02-08 00:02:21 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch to use repoquery --installroot (1.75 KB, patch)
2013-07-29 16:54 EDT, Orion Poplawski
no flags Details | Diff
Patch to use repoquery --installroot (1.75 KB, patch)
2013-07-29 16:56 EDT, Orion Poplawski
no flags Details | Diff
patch to make package state plugin use new repoquery --installroot option (2.91 KB, patch)
2014-01-04 14:57 EST, Clark Williams
no flags Details | Diff

  None (edit)
Description Orion Poplawski 2013-07-18 00:43:30 EDT
Description of problem:

Seeing this on a test koji system with a single builder.  Both i386 and x86_64 builds are kicked off at the same time.  One will usually fail at the repoquery step with:

DEBUG util.py:314:  Executing command: /usr/bin/repoquery -c /var/lib/mock/fedora-rawhide-i386/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/fedora-rawhide-i386/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
DEBUG util.py:264:  Existing lock /var/run/yum.pid: another copy is running as pid 9305.
DEBUG util.py:354:  Child return code was: 50

Version-Release number of selected component (if applicable):
mock-1.1.32-1.fc19.noarch

How reproducible:
Pretty often when two identical builds are started at the same time.

Some thoughts:
- Should repoquery be locking at all with /var/run/yum.pid?
- If it does normally, perhaps it doesn't if -C is used, and perhaps it should?
- Do we have to run it in the chroot?
Comment 1 Orion Poplawski 2013-07-18 00:44:50 EDT
Bringing in the yum-utils folks.
Comment 2 Zdeněk Pavlas 2013-07-18 04:06:12 EDT
Yes, locking was added intentionally to fix BZ 969776.  It's necessary since repoquery can update the metadata if it's too old.  Possible solutions are:

a) run "repoquery -C" as user, then it won't download/lock.  Probably a "yum makecache" should be also run once, to make sure the cache is current.

b) add a sleep-and-retry loop to repoquery locking code.

Note that this might block for very long time, and some users may wish to abort instead, so maybe c) is better:

c) handle the return code 50 in a wrapper and retry.

Do you think b) should be implemented?  Otherwise, it's not a bug.
Comment 3 Orion Poplawski 2013-07-18 12:33:13 EDT
For a), mock has just installed and updated the buildroot with yum so the cache should already be fresh.  However, even with -C locking is still done and the process exits immediately.  I could see locking still being needed so that the repodata isn't updated underneath repoquery.

For b), I'm not sure what the standard behavior for yum these days is.  It might be helpful here.

However, more to the point I think, why isn't this an issue for all of the yum operations that mock does?  Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache.  Shouldn't repoquery be run in the same way?  Can it?
Comment 4 Zdeněk Pavlas 2013-07-19 04:51:09 EDT
> even with -C locking is still done and the process exits immediately.

Not when uid != 0. The intent was that user processes may use root's metadata, without blocking the root (that always does proper locking), at the (small) cost of a possible race.  Maybe we could change the locking code so when *root* runs -C, we don't lock either.

> I'm not sure what the standard behavior for yum these days is.

When Yum can't lock the cachedir it shows the current lock owner, sleeps 2 seconds, and retries.  If it takes too long, users may abort with ctrl-c.

> However, more to the point I think, why isn't this an issue for all of the yum operations that mock does?

Probably due to the above.  But Yum is (mostly) interactive tool, and most of yum-utils are not, so we should not pollute it's output, and just exit with an error.

> Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache.  Shouldn't repoquery be run in the same way?  Can it?

Repoquery can't use custom cachedir. it's mostly just an alternative to "yum search" with more options and output that's easier to parse.
Comment 5 Orion Poplawski 2013-07-19 11:50:08 EDT
(In reply to Zdeněk Pavlas from comment #4)
> > Presumably because it uses a yum cache in /var/cache/mock/<config>/yum_cache.  Shouldn't repoquery be run in the same way?  Can it?
> 
> Repoquery can't use custom cachedir. it's mostly just an alternative to "yum
> search" with more options and output that's easier to parse.

I think we're going to need to be able to do this.  It looks like the way mock does this with yum is to mount /var/cache/mock/<config>/yum_cache into the the install root as /var/cache/yum, then runs yum with --installroot set, which then appears to cause the cache dir to be based off of that.

So, I think repoquery is going to need to either accept the --installroot option or some other way to specify the cache directory.

mock folks, please chime in...
Comment 6 Zdeněk Pavlas 2013-07-22 04:45:25 EDT
> Repoquery can't use custom cachedir

I was wrong.. Right, there's no --installroot repoquery option, but you can make a custom yum.conf, set installroot there, and use repoquery --config option.  Adding --installroot option would not be that hard, too.
Comment 7 Orion Poplawski 2013-07-22 11:20:35 EDT
--installroot would probably be the easiest for mock to support - since it already uses that and the yum.conf that is uses already refer to /var/cache/root (inside the install root).
Comment 8 Jan Zeleny 2013-07-25 03:54:14 EDT
*** Bug 988053 has been marked as a duplicate of this bug. ***
Comment 9 Zdeněk Pavlas 2013-07-25 07:12:27 EDT
So, so I went for the easiest solution.. added a sleep-and-retry loop to repoquery locking code. Repoquery should poll the yum.pid lock instead of failing immediately.
Comment 10 Orion Poplawski 2013-07-25 11:00:53 EDT
While that is helpful, for mock we really need a way to specify what cache directory repoquery uses.  As it stand the yum cache data for *every* different mock configuration (i.e. many different OSes and versions) gets dumped into the /var/cache/yum directory of the mock host.  This is clearly wrong.  Especially when the cache data is already available in /var/cache/mock/<chroot>/yum_cache.
Comment 11 Orion Poplawski 2013-07-29 16:54:14 EDT
Created attachment 780136 [details]
Patch to use repoquery --installroot

So, here's my attempt at making use of the installroot option and -C options.  Seems to be working for me.

I'm assuming that the old _installedPreBuildHook code was creating the tmp mock file to work around the lack on an --installroot option - I'm not sure if that worked or not.

I don't know why the old _availablePostYumHook code didn't use the same trick.

It seems worth it to me to use the -C option since we have just done an install and/or update.
Comment 12 Orion Poplawski 2013-07-29 16:56:15 EDT
Created attachment 780137 [details]
Patch to use repoquery --installroot

Sorry, messed that one up.
Comment 13 Orion Poplawski 2013-07-29 17:16:03 EDT
Hmm, with -C in the _availablePostYumHook I sometimes get messages like:

DEBUG util.py:264:  Cannot find repomd.xml file for rawhide/x86_64

Not sure why yum and repoquery appear to be out of sync here or who the repomd.xml isn't saved.  But perhaps we can't use -C there.
Comment 14 Zdeněk Pavlas 2013-07-30 03:57:39 EDT
Can't tell.. but the code that raises the "Cannot find repomd.xml" exception seems to be correct.  With -C we should not download metadata, and there's definitely no repomd.xml in the cachedir, so we have to bail out.  (the actual cachedir should be inside the installroot, but I might have missed something in http://lists.baseurl.org/pipermail/yum-devel/2013-July/010279.html if you can reproduce and strace this, please check)
Comment 15 Clark Williams 2013-10-23 16:51:36 EDT
Am I correct in presuming that this is going to end up as:

1) modify repoquery/yum with some new options for specifying chroot location
2) teach mock to use these new options
Comment 16 Zdeněk Pavlas 2013-10-24 03:35:14 EDT
The primary problem was that repoquery uses metadata read/write, but it didn't lock at all. Then it did lock, but handled failure with immediate abort, and users expect it to retry.  The --installroot option is (AIUI) mostly just UI stuff that simplifies some repoquery use cases.
Comment 17 Orion Poplawski 2013-12-09 17:54:30 EST
I think the primary problem was that repoquery assumed you were always operating on the system metadata, and didn't have a way to operate on metadata in a different directory.

Clark - That seem correct. 

Any progress here?
Comment 18 Clark Williams 2013-12-13 15:42:01 EST
If I'm interpreting things correctly, repoquery is getting an --installroot option and once that happens I'll modify the package_state plugin for mock to use that and we'll all be happy. Or maybe just not as grumpy...
Comment 19 Orion Poplawski 2013-12-13 15:55:34 EST
Already there:

* Wed Aug 07 2013 Zdenek Pavlas <zpavlas@redhat.com> - 1.1.31-17
- Update to latest HEAD
- Use new findRepos() API for yum-config-manager. BZ 971599
- repoquery: add --installroot option. BZ 988429
Comment 20 Clark Williams 2014-01-04 14:57:56 EST
Created attachment 845508 [details]
patch to make package state plugin use new repoquery --installroot option

Ugh, I rolled this patch in before seeing that Orion had added a patch that does the same thing. Sorry, would have used it if I'd seen it :(

Anyway, this passes all my unit-tests, so it's in for the 1.1.36 build, which should come out next week.
Comment 21 Fedora Update System 2014-02-05 21:08:57 EST
mock-1.1.36-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.36-1.fc19
Comment 22 Fedora Update System 2014-02-05 21:10:40 EST
mock-1.1.36-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.1.36-1.fc20
Comment 23 Fedora Update System 2014-02-05 21:12:10 EST
mock-1.1.36-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.36-1.el6
Comment 24 Fedora Update System 2014-02-06 16:08:07 EST
Package mock-1.1.36-1.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing mock-1.1.36-1.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2014-0467/mock-1.1.36-1.el6
then log in and leave karma (feedback).
Comment 25 Fedora Update System 2014-02-08 00:02:21 EST
mock-1.1.36-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 26 Fedora Update System 2014-03-25 16:24:47 EDT
mock-1.1.37-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.37-1.fc19
Comment 27 Fedora Update System 2014-03-25 16:27:16 EDT
mock-1.1.37-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.1.37-1.fc20
Comment 28 Fedora Update System 2014-03-25 16:29:19 EDT
mock-1.1.37-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.37-1.el6
Comment 29 Fedora Update System 2014-03-27 13:47:50 EDT
mock-1.1.37-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.1.37-2.fc20
Comment 30 Fedora Update System 2014-03-27 13:49:47 EDT
mock-1.1.37-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.37-2.fc19
Comment 31 Fedora Update System 2014-03-27 13:51:37 EDT
mock-1.1.37-2.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.37-2.el6
Comment 32 Fedora Update System 2014-03-31 15:04:26 EDT
mock-1.1.38-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.38-1.fc19
Comment 33 Fedora Update System 2014-03-31 15:06:51 EDT
mock-1.1.38-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.38-1.el6
Comment 34 Fedora Update System 2014-03-31 15:08:44 EDT
mock-1.1.38-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.1.38-1.fc20
Comment 35 Fedora Update System 2014-04-09 09:18:42 EDT
mock-1.1.38-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 36 Fedora Update System 2014-04-18 11:37:39 EDT
mock-1.1.38-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 37 Fedora Update System 2014-04-19 05:19:58 EDT
mock-1.1.38-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.