+++ This bug was initially created as a clone of Bug #982043 +++

Description of problem:
I was attempting to build packages using mock when an error got thrown.  Following was this crash I am now reporting.  The error that was thrown during packaging was the following from yum-utils:

Start: Outputting list of installed packages
ERROR: Exception(/home/dcpyle/rpmbuild/SRPMS/beesu-2.7-12.fc19.src.rpm) Config(fedora-19-i386) 2 minutes 16 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-19-i386/result
ERROR: Command failed. See logs for output.
 # /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{yumdb_info.from_repo}' > /var/lib/mock/fedora-19-i386/result/installed_pkgs

The file referenced above in the error caused by yum-utils was a zero byte file.

Version-Release number of selected component:
yum-utils-1.1.31-15.fc19

Additional info:
reporter:       libreport-2.1.5
cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{yumdb_info.from_repo}'
dso_list:       yum-3.4.3-99.fc19.noarch
executable:     /usr/bin/repoquery
kernel:         3.9.9-301.fc19.x86_64
runlevel:       N 5
uid:            1000

Truncated backtrace:
config.py:1187:_getsysver:YumBaseError: Error: rpmdb open failed

Traceback (most recent call last):
  File "/usr/bin/repoquery", line 1535, in <module>
    main(sys.argv)
  File "/usr/bin/repoquery", line 1411, in main
    repoq.conf
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1054, in <lambda>
    conf = property(fget=lambda self: self._getConfig(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 344, in _getConfig
    startupconf = config.readStartupConfig(fn, root, releasever)
  File "/usr/lib/python2.7/site-packages/yum/config.py", line 1036, in readStartupConfig
    startupconf.distroverpkg)
  File "/usr/lib/python2.7/site-packages/yum/config.py", line 1187, in _getsysver
    raise Errors.YumBaseError("Error: " + str(e))
YumBaseError: Error: rpmdb open failed

Local variables in innermost frame:
installroot: '/var/lib/mock/fedora-19-i386/root/'
e: error('rpmdb open failed',)
ts: <rpmUtils.transaction.TransactionWrapper instance at 0x29c9290>
distroverpkg: 'redhat-release'

--- Additional comment from D. Charles Pyle on 2013-07-08 02:51:54 CEST ---



--- Additional comment from D. Charles Pyle on 2013-07-08 02:51:59 CEST ---



--- Additional comment from D. Charles Pyle on 2013-07-08 02:52:02 CEST ---



--- Additional comment from D. Charles Pyle on 2013-07-08 15:45:22 CEST ---

Another interesting twist.  I just downgraded yum-utils and rebooted.  I tried to build packages again and got the same error.  This time it still is being reported against the above version rather than the version to which I downgraded.  That is particularly strange behavior considering that rpm -qa reports that the version number is one lower than that reported.

$ rpm -qa yum-utils
yum-utils-1.1.31-14.fc19.noarch

--- Additional comment from D. Charles Pyle on 2013-07-08 16:40:42 CEST ---

Tried building packages from other source rpms.  Same errors and results as before with any attempt on any sources.

--- Additional comment from D. Charles Pyle on 2013-07-08 16:42:49 CEST ---

(In reply to D. Charles Pyle from comment #5)
> Tried building packages from other source rpms.  Same errors and results as
> before with any attempt on any sources.

Everything was working last week.  No other changes have been made but several rounds of updates so I am not sure what is going on now.  I have reinstalled packages involving rpm, yum, yum-utils, and mock, and this has not helped the situation.

--- Additional comment from D. Charles Pyle on 2013-07-08 17:10:59 CEST ---

mock --clean
mock --scrub all
mock --init

All make no difference.  The problem persists.  I am going to try with a previous kernel.  Current installed kernel is as follows:

kernel-3.9.9-301.fc19.x86_64

--- Additional comment from D. Charles Pyle on 2013-07-08 17:16:36 CEST ---

Switching kernels makes no difference.  The problem persists.

--- Additional comment from D. Charles Pyle on 2013-07-09 04:05:47 CEST ---

Some more detailed debug info relating to the error above and where it is failing:

Start: Outputting list of installed packages
DEBUG: Executing command: /usr/bin/repoquery -c /tmp/tmp3XiDxf --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{yumdb_info.from_repo}' > /var/lib/mock/fedora-19-i386/result/installed_pkgs with env {'LANG': 'en_US.utf8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG: error: cannot open Packages database in /var/lib/mock/fedora-19-i386/root/var/lib/rpm
DEBUG: Traceback (most recent call last):
DEBUG:   File "/usr/bin/repoquery", line 1535, in <module>
DEBUG:     main(sys.argv)
DEBUG:   File "/usr/bin/repoquery", line 1411, in main
DEBUG:     repoq.conf
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1054, in <lambda>
DEBUG:     conf = property(fget=lambda self: self._getConfig(),
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 344, in _getConfig
DEBUG:     startupconf = config.readStartupConfig(fn, root, releasever)
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/config.py", line 1036, in readStartupConfig
DEBUG:     startupconf.distroverpkg)
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/config.py", line 1187, in _getsysver
DEBUG:     raise Errors.YumBaseError("Error: " + str(e))
DEBUG: yum.Errors.YumBaseError: Error: rpmdb open failed
DEBUG: Child return code was: 1

I am a member of the mock group and root is my primary group.  Nothing has been changed from what was configured as it was before.  I have also rebuilt the databases several times with no effect.

--- Additional comment from D. Charles Pyle on 2013-07-09 04:33:06 CEST ---

Comment 9 is of interest in light of the following additional input:

DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG: error: cannot open Packages database in /var/lib/rpm
DEBUG: Updating / installing...
DEBUG: beesu-2.7-12.fc19                     ########################################
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root

Also present in the rest of the output is the statement that the mockbuild group does not exist.  It does.  I open up users and groups and the mock group is checked for the dcpyle user.  The mock group exists also.

--- Additional comment from D. Charles Pyle on 2013-07-09 06:16:54 CEST ---

Here is something I am seeing from systemd:

bus_connection_get_unix_fd failed Broken pipe
Failed to get caller's security context on: Broken pipe

Might this have something to do with the newfound problems with mock that were not present just over a week ago?

--- Additional comment from D. Charles Pyle on 2013-07-09 08:12:33 CEST ---

The relevant portion of the console output connected with comment 11:

$ journalctl -xb | grep pipe
Jul 08 21:37:25 Illuminatus-1 kernel: SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
Jul 08 21:50:35 Illuminatus-1 systemd[1]: bus_connection_get_unix_fd failed Broken pipe
Jul 08 21:50:35 Illuminatus-1 systemd[1]: Failed to get caller's security context on: Broken pipe

Could this be caused by an SELinux issue?

--- Additional comment from D. Charles Pyle on 2013-07-09 08:32:01 CEST ---

Just resolved the broken pipe issue by modifying services, switching some off and some off and then on again.  No change in the yum-utils error. The problem described above remains.

--- Additional comment from Zdeněk Pavlas on 2013-07-09 09:51:36 CEST ---

> cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} 

Can you add /tmp/tmpgPkPCR too?  It probably just sets installroot, but to make sure..

> uid:            1000
> installroot: '/var/lib/mock/fedora-19-i386/root/'
> YumBaseError: Error: rpmdb open failed

It seems files in /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ are not readable (they should be 644, but uid 1000 can't open it RDONLY for some reason).  Yum then can't detect --releasever, and repoquery aborts.

> DEBUG: error: cannot open Packages index using db5 - Permission denied (13)

13 == EPERM.

> Could this be caused by an SELinux issue?

I think it's likely.. rpmdb in the installroot is not readable.

--- Additional comment from D. Charles Pyle on 2013-07-09 15:13:30 CEST ---

(In reply to Zdeněk Pavlas from comment #14)
> > cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} 
> 
> Can you add /tmp/tmpgPkPCR too?  It probably just sets installroot, but to
> make sure..
> 

/tmp/tmpgPkPCR is gone missing.  There is another file of the same kind (owned by root:mock), with a different name, however.  The contents of that file are as follows:

[main]
installroot=/var/lib/mock/fedora-19-i386/root/


> > uid:            1000
> > installroot: '/var/lib/mock/fedora-19-i386/root/'
> > YumBaseError: Error: rpmdb open failed
> 
> It seems files in /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ are not
> readable (they should be 644, but uid 1000 can't open it RDONLY for some
> reason).  Yum then can't detect --releasever, and repoquery aborts.
> 
> > DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
> 
> 13 == EPERM.
> 

This is really odd, too.  I cleaned out and even deleted all the files and rebuilt it all using mock -v -r fedora-19-i386 --init on the command line. No change.

Another odd behavior I noticed afterward is that when running the above command, the /var directory is not formed.  I had to run mock -r fedora-19-i386 --shell and rebuild the database.  That failed, so I ran sudo rpm -r /var/lib/mock/fedora-19-i386/root --rebuilddb and then changed the access permissions to the files. That did not work either.  The problem remained.

So, I ran chmod -R 644 rpm as root and again ran mock -v -r fedora-19-i386 --rebuild '/home/dcpyle/rpmbuild/SRPMS/beesu-2.7-12.fc19.src.rpm'.  I saw all sorts of dependency errors and it failed again.  In addition, all the files' permissions were set back to what they were before running the chmod command.

> > Could this be caused by an SELinux issue?
> 
> I think it's likely.. rpmdb in the installroot is not readable.

I set SELinux to permissive and it does this.  I am going to turn SELinux off and see if that makes any difference, although I would have expected setting it to permissive to not affect anything there.

--- Additional comment from D. Charles Pyle on 2013-07-09 15:26:52 CEST ---

Here is the latest debugging output for the last attempt at running the command with the verbose switch on.  This was run with SELinux set to permissive and the contents of the /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ directory contents set to what it is supposed to be before being set back to unreadable by yum.

Notice also in the output that it claims that my user does not exist.  I have verified that my user exists, and that it is part of the mock group.  I'll post a screenshot of Users and Groups in just a moment.

--- Additional comment from D. Charles Pyle on 2013-07-09 15:31:34 CEST ---

As you can see from this screenshot, the user mock claims does not exist does, in fact, exist, and this existing user belongs to both groups mock and mockbuild.  I post this from the existing user.

What do you think is causing mock to think my user does not exist?

--- Additional comment from D. Charles Pyle on 2013-07-09 16:15:45 CEST ---

I just tried again with enforcing=0 and got the same results as in the above comments.  So, I am no longer thinking it is an SELinux issue.  Not sure where the issue lies at the moment.

--- Additional comment from Zdeněk Pavlas on 2013-07-09 16:42:24 CEST ---

Changing the component as repoquery fails because mock (likely) didn't create a valid environment (none or invalid rpmdb). not a Yum bug.

--- Additional comment from Tom Klingenberg on 2013-09-17 11:19:03 CEST ---

I get the error message during each boot since I upgraded from Fedora 17 to Fedora 19.

--- Additional comment from D. Charles Pyle on 2013-10-06 18:24:27 CEST ---

I found a workaround for this error, which still is occurring in F20.

Adding --disable-plugin=package_state to the end of the command line stops the error and allows mock to build the packages without failing.

--- Additional comment from Zdeněk Pavlas on 2013-10-07 10:52:11 CEST ---

Thanks, I have no idea what this plugin is doing, and where it comes from. Please add the "Loaded plugins:" info when reporting bugs esp. when you use anything not-quite-standard (not built from yum-utils package).

package_state: what's that? What's the output of 
"rpm -qf /usr/lib/yum-plugins/package_state.py"?

--- Additional comment from Zdeněk Pavlas on 2013-10-07 11:20:27 CEST ---



--- Additional comment from Zdeněk Pavlas on 2013-10-07 11:28:48 CEST ---

Sorry, found out that package_state is NOT a yum plugin, it's an internal mockbuild plugin. Please ignore the previous comment.

--- Additional comment from D. Charles Pyle on 2013-10-07 14:54:50 CEST ---

The odd thing is that the crash is identified by abrt as being in yum-utils.

--- Additional comment from Zdeněk Pavlas on 2013-10-07 15:09:18 CEST ---

I can get rid of the traceback, but not fix the underlying bug.  The "rpmdb open failed" error is handled by Yum like this:

$ yum --installroot=/foo info info --disablerepo=*
error: cannot open Packages database in /foo/var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

I can change repoquery to print a similar message instead of the traceback, but that won't help you in any way.

--- Additional comment from D. Charles Pyle on 2013-10-07 15:16:53 CEST ---

OK.  I can just use the --disable-plugin=package_state and --disable-plugin=package_state --init workarounds as needed for now.

--- Additional comment from Lars Kellogg-Stedman on 2013-11-01 16:08:00 CET ---

I've just run into the same bug on my Fedora 19 system.  Adding --disable-plugin=package_state to the mock command line resolves it for me, too.

--- Additional comment from Clark Williams on 2013-11-04 19:53:20 CET ---

I see that you're trying to build for f19 but wondered what distro you're using to host the builds?

--- Additional comment from Clark Williams on 2013-11-04 20:07:15 CET ---

BTW, the package_state plugin is primarily used by build systems like koji. If you're just building packages there's no need for you to use it. 

That being said I'd like to get to the bottom of this problem.

--- Additional comment from D. Charles Pyle on 2013-11-05 03:30:01 CET ---

At first, I was building on Fedora 19 x86_64.  Afterward, I continued doing so on a fresh install of Fedora 20 x86_64.

--- Additional comment from Fedora Update System on 2013-12-13 12:45:49 CET ---

yum-utils-1.1.31-19.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/yum-utils-1.1.31-19.fc20

--- Additional comment from Fedora Update System on 2013-12-13 18:54:58 CET ---

Package yum-utils-1.1.31-19.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing yum-utils-1.1.31-19.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23330/yum-utils-1.1.31-19.fc20
then log in and leave karma (feedback).

--- Additional comment from Fedora Update System on 2013-12-16 08:05:52 CET ---

yum-utils-1.1.31-19.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.