Bug 982043

Created attachment 770209 [details]
File: backtrace

Created attachment 770210 [details]
File: core_backtrace

Created attachment 770211 [details]
File: environ

Another interesting twist.  I just downgraded yum-utils and rebooted.  I tried to build packages again and got the same error.  This time it still is being reported against the above version rather than the version to which I downgraded.  That is particularly strange behavior considering that rpm -qa reports that the version number is one lower than that reported.

$ rpm -qa yum-utils
yum-utils-1.1.31-14.fc19.noarch

Tried building packages from other source rpms.  Same errors and results as before with any attempt on any sources.

(In reply to D. Charles Pyle from comment #5)
> Tried building packages from other source rpms.  Same errors and results as
> before with any attempt on any sources.

Everything was working last week.  No other changes have been made but several rounds of updates so I am not sure what is going on now.  I have reinstalled packages involving rpm, yum, yum-utils, and mock, and this has not helped the situation.

mock --clean
mock --scrub all
mock --init

All make no difference.  The problem persists.  I am going to try with a previous kernel.  Current installed kernel is as follows:

kernel-3.9.9-301.fc19.x86_64

Switching kernels makes no difference.  The problem persists.

Some more detailed debug info relating to the error above and where it is failing:

Start: Outputting list of installed packages
DEBUG: Executing command: /usr/bin/repoquery -c /tmp/tmp3XiDxf --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{yumdb_info.from_repo}' > /var/lib/mock/fedora-19-i386/result/installed_pkgs with env {'LANG': 'en_US.utf8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG: error: cannot open Packages database in /var/lib/mock/fedora-19-i386/root/var/lib/rpm
DEBUG: Traceback (most recent call last):
DEBUG:   File "/usr/bin/repoquery", line 1535, in <module>
DEBUG:     main(sys.argv)
DEBUG:   File "/usr/bin/repoquery", line 1411, in main
DEBUG:     repoq.conf
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1054, in <lambda>
DEBUG:     conf = property(fget=lambda self: self._getConfig(),
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 344, in _getConfig
DEBUG:     startupconf = config.readStartupConfig(fn, root, releasever)
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/config.py", line 1036, in readStartupConfig
DEBUG:     startupconf.distroverpkg)
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/config.py", line 1187, in _getsysver
DEBUG:     raise Errors.YumBaseError("Error: " + str(e))
DEBUG: yum.Errors.YumBaseError: Error: rpmdb open failed
DEBUG: Child return code was: 1

I am a member of the mock group and root is my primary group.  Nothing has been changed from what was configured as it was before.  I have also rebuilt the databases several times with no effect.

Comment 9 is of interest in light of the following additional input:

DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG: error: cannot open Packages database in /var/lib/rpm
DEBUG: Updating / installing...
DEBUG: beesu-2.7-12.fc19                     ########################################
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root

Also present in the rest of the output is the statement that the mockbuild group does not exist.  It does.  I open up users and groups and the mock group is checked for the dcpyle user.  The mock group exists also.

Here is something I am seeing from systemd:

bus_connection_get_unix_fd failed Broken pipe
Failed to get caller's security context on: Broken pipe

Might this have something to do with the newfound problems with mock that were not present just over a week ago?

The relevant portion of the console output connected with comment 11:

$ journalctl -xb | grep pipe
Jul 08 21:37:25 Illuminatus-1 kernel: SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
Jul 08 21:50:35 Illuminatus-1 systemd[1]: bus_connection_get_unix_fd failed Broken pipe
Jul 08 21:50:35 Illuminatus-1 systemd[1]: Failed to get caller's security context on: Broken pipe

Could this be caused by an SELinux issue?

Just resolved the broken pipe issue by modifying services, switching some off and some off and then on again.  No change in the yum-utils error. The problem described above remains.

> cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} 

Can you add /tmp/tmpgPkPCR too?  It probably just sets installroot, but to make sure..

> uid:            1000
> installroot: '/var/lib/mock/fedora-19-i386/root/'
> YumBaseError: Error: rpmdb open failed

It seems files in /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ are not readable (they should be 644, but uid 1000 can't open it RDONLY for some reason).  Yum then can't detect --releasever, and repoquery aborts.

> DEBUG: error: cannot open Packages index using db5 - Permission denied (13)

13 == EPERM.

> Could this be caused by an SELinux issue?

I think it's likely.. rpmdb in the installroot is not readable.

(In reply to Zdeněk Pavlas from comment #14)
> > cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} 
> 
> Can you add /tmp/tmpgPkPCR too?  It probably just sets installroot, but to
> make sure..
> 

/tmp/tmpgPkPCR is gone missing.  There is another file of the same kind (owned by root:mock), with a different name, however.  The contents of that file are as follows:

[main]
installroot=/var/lib/mock/fedora-19-i386/root/


> > uid:            1000
> > installroot: '/var/lib/mock/fedora-19-i386/root/'
> > YumBaseError: Error: rpmdb open failed
> 
> It seems files in /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ are not
> readable (they should be 644, but uid 1000 can't open it RDONLY for some
> reason).  Yum then can't detect --releasever, and repoquery aborts.
> 
> > DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
> 
> 13 == EPERM.
> 

This is really odd, too.  I cleaned out and even deleted all the files and rebuilt it all using mock -v -r fedora-19-i386 --init on the command line. No change.

Another odd behavior I noticed afterward is that when running the above command, the /var directory is not formed.  I had to run mock -r fedora-19-i386 --shell and rebuild the database.  That failed, so I ran sudo rpm -r /var/lib/mock/fedora-19-i386/root --rebuilddb and then changed the access permissions to the files. That did not work either.  The problem remained.

So, I ran chmod -R 644 rpm as root and again ran mock -v -r fedora-19-i386 --rebuild '/home/dcpyle/rpmbuild/SRPMS/beesu-2.7-12.fc19.src.rpm'.  I saw all sorts of dependency errors and it failed again.  In addition, all the files' permissions were set back to what they were before running the chmod command.

> > Could this be caused by an SELinux issue?
> 
> I think it's likely.. rpmdb in the installroot is not readable.

I set SELinux to permissive and it does this.  I am going to turn SELinux off and see if that makes any difference, although I would have expected setting it to permissive to not affect anything there.

Created attachment 771017 [details]
Full output of latest attempt with mock.

Here is the latest debugging output for the last attempt at running the command with the verbose switch on.  This was run with SELinux set to permissive and the contents of the /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ directory contents set to what it is supposed to be before being set back to unreadable by yum.

Notice also in the output that it claims that my user does not exist.  I have verified that my user exists, and that it is part of the mock group.  I'll post a screenshot of Users and Groups in just a moment.

Created attachment 771021 [details]
Screenshot of relevant groups and existing username.

As you can see from this screenshot, the user mock claims does not exist does, in fact, exist, and this existing user belongs to both groups mock and mockbuild.  I post this from the existing user.

What do you think is causing mock to think my user does not exist?

I just tried again with enforcing=0 and got the same results as in the above comments.  So, I am no longer thinking it is an SELinux issue.  Not sure where the issue lies at the moment.

Changing the component as repoquery fails because mock (likely) didn't create a valid environment (none or invalid rpmdb). not a Yum bug.

I get the error message during each boot since I upgraded from Fedora 17 to Fedora 19.

I found a workaround for this error, which still is occurring in F20.

Adding --disable-plugin=package_state to the end of the command line stops the error and allows mock to build the packages without failing.

Thanks, I have no idea what this plugin is doing, and where it comes from. Please add the "Loaded plugins:" info when reporting bugs esp. when you use anything not-quite-standard (not built from yum-utils package).

package_state: what's that? What's the output of 
"rpm -qf /usr/lib/yum-plugins/package_state.py"?

*** Bug 1015899 has been marked as a duplicate of this bug. ***

Sorry, found out that package_state is NOT a yum plugin, it's an internal mockbuild plugin. Please ignore the previous comment.

The odd thing is that the crash is identified by abrt as being in yum-utils.

I can get rid of the traceback, but not fix the underlying bug.  The "rpmdb open failed" error is handled by Yum like this:

$ yum --installroot=/foo info info --disablerepo=*
error: cannot open Packages database in /foo/var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

I can change repoquery to print a similar message instead of the traceback, but that won't help you in any way.

OK.  I can just use the --disable-plugin=package_state and --disable-plugin=package_state --init workarounds as needed for now.

I've just run into the same bug on my Fedora 19 system.  Adding --disable-plugin=package_state to the mock command line resolves it for me, too.

I see that you're trying to build for f19 but wondered what distro you're using to host the builds?

BTW, the package_state plugin is primarily used by build systems like koji. If you're just building packages there's no need for you to use it. 

That being said I'd like to get to the bottom of this problem.

At first, I was building on Fedora 19 x86_64.  Afterward, I continued doing so on a fresh install of Fedora 20 x86_64.

yum-utils-1.1.31-19.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/yum-utils-1.1.31-19.fc20

Package yum-utils-1.1.31-19.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing yum-utils-1.1.31-19.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23330/yum-utils-1.1.31-19.fc20
then log in and leave karma (feedback).

yum-utils-1.1.31-19.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.