Bug 982043 - [abrt] yum-utils-1.1.31-15.fc19: config.py:1187:_getsysver:YumBaseError: Error: rpmdb open failed
[abrt] yum-utils-1.1.31-15.fc19: config.py:1187:_getsysver:YumBaseError: Erro...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: mock (Show other bugs)
19
x86_64 Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Clark Williams
Fedora Extras Quality Assurance
abrt_hash:86506b4195ebca86a90a629882e...
:
: 1015899 (view as bug list)
Depends On:
Blocks: 1061486
  Show dependency treegraph
 
Reported: 2013-07-07 20:51 EDT by D. Charles Pyle
Modified: 2014-02-04 17:33 EST (History)
8 users (show)

See Also:
Fixed In Version: yum-utils-1.1.31-19.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1061486 (view as bug list)
Environment:
Last Closed: 2013-12-16 02:05:52 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: backtrace (1.00 KB, text/plain)
2013-07-07 20:51 EDT, D. Charles Pyle
no flags Details
File: core_backtrace (587 bytes, text/plain)
2013-07-07 20:51 EDT, D. Charles Pyle
no flags Details
File: environ (153 bytes, text/plain)
2013-07-07 20:52 EDT, D. Charles Pyle
no flags Details
Full output of latest attempt with mock. (57.18 KB, text/plain)
2013-07-09 09:26 EDT, D. Charles Pyle
no flags Details
Screenshot of relevant groups and existing username. (56.23 KB, image/png)
2013-07-09 09:31 EDT, D. Charles Pyle
no flags Details

  None (edit)
Description D. Charles Pyle 2013-07-07 20:51:50 EDT
Description of problem:
I was attempting to build packages using mock when an error got thrown.  Following was this crash I am now reporting.  The error that was thrown during packaging was the following from yum-utils:

Start: Outputting list of installed packages
ERROR: Exception(/home/dcpyle/rpmbuild/SRPMS/beesu-2.7-12.fc19.src.rpm) Config(fedora-19-i386) 2 minutes 16 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-19-i386/result
ERROR: Command failed. See logs for output.
 # /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{yumdb_info.from_repo}' > /var/lib/mock/fedora-19-i386/result/installed_pkgs

The file referenced above in the error caused by yum-utils was a zero byte file.

Version-Release number of selected component:
yum-utils-1.1.31-15.fc19

Additional info:
reporter:       libreport-2.1.5
cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{yumdb_info.from_repo}'
dso_list:       yum-3.4.3-99.fc19.noarch
executable:     /usr/bin/repoquery
kernel:         3.9.9-301.fc19.x86_64
runlevel:       N 5
uid:            1000

Truncated backtrace:
config.py:1187:_getsysver:YumBaseError: Error: rpmdb open failed

Traceback (most recent call last):
  File "/usr/bin/repoquery", line 1535, in <module>
    main(sys.argv)
  File "/usr/bin/repoquery", line 1411, in main
    repoq.conf
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1054, in <lambda>
    conf = property(fget=lambda self: self._getConfig(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 344, in _getConfig
    startupconf = config.readStartupConfig(fn, root, releasever)
  File "/usr/lib/python2.7/site-packages/yum/config.py", line 1036, in readStartupConfig
    startupconf.distroverpkg)
  File "/usr/lib/python2.7/site-packages/yum/config.py", line 1187, in _getsysver
    raise Errors.YumBaseError("Error: " + str(e))
YumBaseError: Error: rpmdb open failed

Local variables in innermost frame:
installroot: '/var/lib/mock/fedora-19-i386/root/'
e: error('rpmdb open failed',)
ts: <rpmUtils.transaction.TransactionWrapper instance at 0x29c9290>
distroverpkg: 'redhat-release'
Comment 1 D. Charles Pyle 2013-07-07 20:51:54 EDT
Created attachment 770209 [details]
File: backtrace
Comment 2 D. Charles Pyle 2013-07-07 20:51:59 EDT
Created attachment 770210 [details]
File: core_backtrace
Comment 3 D. Charles Pyle 2013-07-07 20:52:02 EDT
Created attachment 770211 [details]
File: environ
Comment 4 D. Charles Pyle 2013-07-08 09:45:22 EDT
Another interesting twist.  I just downgraded yum-utils and rebooted.  I tried to build packages again and got the same error.  This time it still is being reported against the above version rather than the version to which I downgraded.  That is particularly strange behavior considering that rpm -qa reports that the version number is one lower than that reported.

$ rpm -qa yum-utils
yum-utils-1.1.31-14.fc19.noarch
Comment 5 D. Charles Pyle 2013-07-08 10:40:42 EDT
Tried building packages from other source rpms.  Same errors and results as before with any attempt on any sources.
Comment 6 D. Charles Pyle 2013-07-08 10:42:49 EDT
(In reply to D. Charles Pyle from comment #5)
> Tried building packages from other source rpms.  Same errors and results as
> before with any attempt on any sources.

Everything was working last week.  No other changes have been made but several rounds of updates so I am not sure what is going on now.  I have reinstalled packages involving rpm, yum, yum-utils, and mock, and this has not helped the situation.
Comment 7 D. Charles Pyle 2013-07-08 11:10:59 EDT
mock --clean
mock --scrub all
mock --init

All make no difference.  The problem persists.  I am going to try with a previous kernel.  Current installed kernel is as follows:

kernel-3.9.9-301.fc19.x86_64
Comment 8 D. Charles Pyle 2013-07-08 11:16:36 EDT
Switching kernels makes no difference.  The problem persists.
Comment 9 D. Charles Pyle 2013-07-08 22:05:47 EDT
Some more detailed debug info relating to the error above and where it is failing:

Start: Outputting list of installed packages
DEBUG: Executing command: /usr/bin/repoquery -c /tmp/tmp3XiDxf --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{yumdb_info.from_repo}' > /var/lib/mock/fedora-19-i386/result/installed_pkgs with env {'LANG': 'en_US.utf8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG: error: cannot open Packages database in /var/lib/mock/fedora-19-i386/root/var/lib/rpm
DEBUG: Traceback (most recent call last):
DEBUG:   File "/usr/bin/repoquery", line 1535, in <module>
DEBUG:     main(sys.argv)
DEBUG:   File "/usr/bin/repoquery", line 1411, in main
DEBUG:     repoq.conf
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1054, in <lambda>
DEBUG:     conf = property(fget=lambda self: self._getConfig(),
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 344, in _getConfig
DEBUG:     startupconf = config.readStartupConfig(fn, root, releasever)
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/config.py", line 1036, in readStartupConfig
DEBUG:     startupconf.distroverpkg)
DEBUG:   File "/usr/lib/python2.7/site-packages/yum/config.py", line 1187, in _getsysver
DEBUG:     raise Errors.YumBaseError("Error: " + str(e))
DEBUG: yum.Errors.YumBaseError: Error: rpmdb open failed
DEBUG: Child return code was: 1

I am a member of the mock group and root is my primary group.  Nothing has been changed from what was configured as it was before.  I have also rebuilt the databases several times with no effect.
Comment 10 D. Charles Pyle 2013-07-08 22:33:06 EDT
Comment 9 is of interest in light of the following additional input:

DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG: error: cannot open Packages database in /var/lib/rpm
DEBUG: Updating / installing...
DEBUG: beesu-2.7-12.fc19                     ########################################
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root
DEBUG: warning: user dcpyle does not exist - using root

Also present in the rest of the output is the statement that the mockbuild group does not exist.  It does.  I open up users and groups and the mock group is checked for the dcpyle user.  The mock group exists also.
Comment 11 D. Charles Pyle 2013-07-09 00:16:54 EDT
Here is something I am seeing from systemd:

bus_connection_get_unix_fd failed Broken pipe
Failed to get caller's security context on: Broken pipe

Might this have something to do with the newfound problems with mock that were not present just over a week ago?
Comment 12 D. Charles Pyle 2013-07-09 02:12:33 EDT
The relevant portion of the console output connected with comment 11:

$ journalctl -xb | grep pipe
Jul 08 21:37:25 Illuminatus-1 kernel: SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
Jul 08 21:50:35 Illuminatus-1 systemd[1]: bus_connection_get_unix_fd failed Broken pipe
Jul 08 21:50:35 Illuminatus-1 systemd[1]: Failed to get caller's security context on: Broken pipe

Could this be caused by an SELinux issue?
Comment 13 D. Charles Pyle 2013-07-09 02:32:01 EDT
Just resolved the broken pipe issue by modifying services, switching some off and some off and then on again.  No change in the yum-utils error. The problem described above remains.
Comment 14 Zdeněk Pavlas 2013-07-09 03:51:36 EDT
> cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} 

Can you add /tmp/tmpgPkPCR too?  It probably just sets installroot, but to make sure..

> uid:            1000
> installroot: '/var/lib/mock/fedora-19-i386/root/'
> YumBaseError: Error: rpmdb open failed

It seems files in /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ are not readable (they should be 644, but uid 1000 can't open it RDONLY for some reason).  Yum then can't detect --releasever, and repoquery aborts.

> DEBUG: error: cannot open Packages index using db5 - Permission denied (13)

13 == EPERM.

> Could this be caused by an SELinux issue?

I think it's likely.. rpmdb in the installroot is not readable.
Comment 15 D. Charles Pyle 2013-07-09 09:13:30 EDT
(In reply to Zdeněk Pavlas from comment #14)
> > cmdline:        /usr/bin/python -tt /usr/bin/repoquery -c /tmp/tmpgPkPCR --installed -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} 
> 
> Can you add /tmp/tmpgPkPCR too?  It probably just sets installroot, but to
> make sure..
> 

/tmp/tmpgPkPCR is gone missing.  There is another file of the same kind (owned by root:mock), with a different name, however.  The contents of that file are as follows:

[main]
installroot=/var/lib/mock/fedora-19-i386/root/


> > uid:            1000
> > installroot: '/var/lib/mock/fedora-19-i386/root/'
> > YumBaseError: Error: rpmdb open failed
> 
> It seems files in /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ are not
> readable (they should be 644, but uid 1000 can't open it RDONLY for some
> reason).  Yum then can't detect --releasever, and repoquery aborts.
> 
> > DEBUG: error: cannot open Packages index using db5 - Permission denied (13)
> 
> 13 == EPERM.
> 

This is really odd, too.  I cleaned out and even deleted all the files and rebuilt it all using mock -v -r fedora-19-i386 --init on the command line. No change.

Another odd behavior I noticed afterward is that when running the above command, the /var directory is not formed.  I had to run mock -r fedora-19-i386 --shell and rebuild the database.  That failed, so I ran sudo rpm -r /var/lib/mock/fedora-19-i386/root --rebuilddb and then changed the access permissions to the files. That did not work either.  The problem remained.

So, I ran chmod -R 644 rpm as root and again ran mock -v -r fedora-19-i386 --rebuild '/home/dcpyle/rpmbuild/SRPMS/beesu-2.7-12.fc19.src.rpm'.  I saw all sorts of dependency errors and it failed again.  In addition, all the files' permissions were set back to what they were before running the chmod command.

> > Could this be caused by an SELinux issue?
> 
> I think it's likely.. rpmdb in the installroot is not readable.

I set SELinux to permissive and it does this.  I am going to turn SELinux off and see if that makes any difference, although I would have expected setting it to permissive to not affect anything there.
Comment 16 D. Charles Pyle 2013-07-09 09:26:52 EDT
Created attachment 771017 [details]
Full output of latest attempt with mock.

Here is the latest debugging output for the last attempt at running the command with the verbose switch on.  This was run with SELinux set to permissive and the contents of the /var/lib/mock/fedora-19-i386/root/var/lib/rpm/ directory contents set to what it is supposed to be before being set back to unreadable by yum.

Notice also in the output that it claims that my user does not exist.  I have verified that my user exists, and that it is part of the mock group.  I'll post a screenshot of Users and Groups in just a moment.
Comment 17 D. Charles Pyle 2013-07-09 09:31:34 EDT
Created attachment 771021 [details]
Screenshot of relevant groups and existing username.

As you can see from this screenshot, the user mock claims does not exist does, in fact, exist, and this existing user belongs to both groups mock and mockbuild.  I post this from the existing user.

What do you think is causing mock to think my user does not exist?
Comment 18 D. Charles Pyle 2013-07-09 10:15:45 EDT
I just tried again with enforcing=0 and got the same results as in the above comments.  So, I am no longer thinking it is an SELinux issue.  Not sure where the issue lies at the moment.
Comment 19 Zdeněk Pavlas 2013-07-09 10:42:24 EDT
Changing the component as repoquery fails because mock (likely) didn't create a valid environment (none or invalid rpmdb). not a Yum bug.
Comment 20 Tom Klingenberg 2013-09-17 05:19:03 EDT
I get the error message during each boot since I upgraded from Fedora 17 to Fedora 19.
Comment 21 D. Charles Pyle 2013-10-06 12:24:27 EDT
I found a workaround for this error, which still is occurring in F20.

Adding --disable-plugin=package_state to the end of the command line stops the error and allows mock to build the packages without failing.
Comment 22 Zdeněk Pavlas 2013-10-07 04:52:11 EDT
Thanks, I have no idea what this plugin is doing, and where it comes from. Please add the "Loaded plugins:" info when reporting bugs esp. when you use anything not-quite-standard (not built from yum-utils package).

package_state: what's that? What's the output of 
"rpm -qf /usr/lib/yum-plugins/package_state.py"?
Comment 23 Zdeněk Pavlas 2013-10-07 05:20:27 EDT
*** Bug 1015899 has been marked as a duplicate of this bug. ***
Comment 24 Zdeněk Pavlas 2013-10-07 05:28:48 EDT
Sorry, found out that package_state is NOT a yum plugin, it's an internal mockbuild plugin. Please ignore the previous comment.
Comment 25 D. Charles Pyle 2013-10-07 08:54:50 EDT
The odd thing is that the crash is identified by abrt as being in yum-utils.
Comment 26 Zdeněk Pavlas 2013-10-07 09:09:18 EDT
I can get rid of the traceback, but not fix the underlying bug.  The "rpmdb open failed" error is handled by Yum like this:

$ yum --installroot=/foo info info --disablerepo=*
error: cannot open Packages database in /foo/var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

I can change repoquery to print a similar message instead of the traceback, but that won't help you in any way.
Comment 27 D. Charles Pyle 2013-10-07 09:16:53 EDT
OK.  I can just use the --disable-plugin=package_state and --disable-plugin=package_state --init workarounds as needed for now.
Comment 28 Lars Kellogg-Stedman 2013-11-01 11:08:00 EDT
I've just run into the same bug on my Fedora 19 system.  Adding --disable-plugin=package_state to the mock command line resolves it for me, too.
Comment 29 Clark Williams 2013-11-04 13:53:20 EST
I see that you're trying to build for f19 but wondered what distro you're using to host the builds?
Comment 30 Clark Williams 2013-11-04 14:07:15 EST
BTW, the package_state plugin is primarily used by build systems like koji. If you're just building packages there's no need for you to use it. 

That being said I'd like to get to the bottom of this problem.
Comment 31 D. Charles Pyle 2013-11-04 21:30:01 EST
At first, I was building on Fedora 19 x86_64.  Afterward, I continued doing so on a fresh install of Fedora 20 x86_64.
Comment 32 Fedora Update System 2013-12-13 06:45:49 EST
yum-utils-1.1.31-19.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/yum-utils-1.1.31-19.fc20
Comment 33 Fedora Update System 2013-12-13 12:54:58 EST
Package yum-utils-1.1.31-19.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing yum-utils-1.1.31-19.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23330/yum-utils-1.1.31-19.fc20
then log in and leave karma (feedback).
Comment 34 Fedora Update System 2013-12-16 02:05:52 EST
yum-utils-1.1.31-19.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.