Bug 1299482

Created attachment 1115852 [details]
root.log

Created attachment 1115853 [details]
state.log

I cleaned out /var/cache/mock and it works again.

I hit the same issue, and cleaning up both /var/lib/mock and /var/cache/mock doesn't fix it.

It seems DNF cannot cope with a non-existing RPM DB anymore. I don't know if it used to create one automatically, but it doesn't anymore. The error message seems to be coming from hawkey, so I guess some unintended behavior change was introduced that broke it.

I cannot reproduce it with mock-1.2.17-1.fc24.noarch

These are the versions of all the software that I think can be relevant in my system:

dnf-0:1.1.9-2.fc23.noarch
python3-dnf-plugins-core-0.1.21-2.fc23.noarch
rpm-0:4.13.0-0.rc1.13.fc23.x86_64
hawkey-0.6.2-3.fc23.x86_64
python3-hawkey-0.6.2-3.fc23.x86_64
librepo-1.7.16-2.fc23.x86_64
python3-librepo-1.7.16-2.fc23.x86_64
libsolv-0.6.20-2.fc23.x86_64

It probably doesn't matter, but my root filesystem is set up with BTRFS on top of LVM.

I'll try to reproduce the issue from a clean install (in a VM) and report back.

Similar problem here (F24 x86_64) Mock starts to fail with this RPMDB checksum error and quick looking at /var/lib/mock/fedora-24-i386/root/var/lib/rpm reveals directory was empty. 

Workaround was: rpmdb --initdb --dbpath=/var/lib/mock/fedora-24-i386/root/var/lib/rpm and using -n switch in next mock invocation.

Can confirm the same finding on fresh Fedora 25 Beta install…

This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Same with mock-1.3.2-1.fc25.noarch.  I frequently have to run rpmdb --initdb manually to get going.  Once there's a root cache, that can be reused just fine, but try --scrub=all and then --init, and it's a problem again.

I'm still facing this issue, and it really breaks my workflow. Do we know what's causing this?

(In reply to David Kaspar [Dee'Kej] from comment #12)
> Do we know
> what's causing this?

No. And personally I cannot reproduce it. VM image or Dockerfile with the reproducer will be helpful.

Well, I'm trying to reproduce it in a VM, but it stubbornly works fine.  :/

I even copied my entire /etc/mock/, so I know that's not the problem.
If I can find the meaningful difference, I'll let you know...

Via strace, I figured out this is the dnf debuginfo plugin.

Edit /etc/dnf/plugins/debuginfo-install.conf, and set autoupdate=1.  Now when mock tries to init, the plugin sees the bogus rpmdb and bombs out.

I can upload a VM image if you need, but I think this is easy to reproduce now.

So I have two workarounds:

1. Set autoupdate=0 for the host.  But that's not desirable, since I do want debuginfo updating.

2. Disable the debuginfo-install plugin via mock's dnf_common_opts in site-defaults.cfg.  This is probably the best option for now.

Hello guys,

I can confirm that currently I have 'autoupdate=1' enabled for debuginfo. I will try next week to reproduce this on my machine, both with 'autoupdate=1' and 'autoupdate=0'.

Best regards,

Dee'Kej

Hitting the same, but in addition to the debuginfo-install plugin I also needed to disable the leaves plugin.

I can confirm that disabling the debuginfo-install plugin fixes the issue. It seems to be sufficient to do it in /etc/mock/site-defaults.cfg by adding `--disableplugin=debuginfo-install` to the DNF command line.

(In reply to Daniel Miranda from comment #19)
> I can confirm that disabling the debuginfo-install plugin fixes the issue.
> It seems to be sufficient to do it in /etc/mock/site-defaults.cfg by adding
> `--disableplugin=debuginfo-install` to the DNF command line.

I feel like this should be added to the default configuration. I spent a good 15-30 minutes figuring out why my mock has stopped working and trying to get it to work again.

(In reply to Susi Lehtola from comment #20)
> I feel like this should be added to the default configuration. I spent a
> good 15-30 minutes figuring out why my mock has stopped working and trying
> to get it to work again.

I agree with you on this. Maybe we should propose this @fedora-devel mailing list?

I have just hit the same issue with F26.
The  workaround for me was to change /etc/mock/site-defaults.cfg
From
# config_opts['dnf_common_opts'] = ['--disableplugin=local', '--setopt=deltarpm=False']
To
config_opts['dnf_common_opts'] = ['--noplugins', '--setopt=deltarpm=False']

I.e. to use --noplugins instead of --disableplugin=local.

new mock user ... Tomas' solution in Comment 22 worked for me.  Thanks.

The problem still persists with:
$ rpm -qa mock rpm dnf*
dnf-conf-2.6.2-1.fc26.noarch
dnf-2.6.2-1.fc26.noarch
dnf-plugins-core-2.1.3-1.fc26.noarch
rpm-4.13.0.1-5.fc26.x86_64
dnf-yum-2.6.2-1.fc26.noarch
mock-1.4.2-1.fc26.noarch

Also /etc/mock/default.cfg is not created on a clean install of Fedora 26, but that's a different issue which is already addressed in multiple duplicate bug reports already.

When I run:
$ mock psad-2.4.4-5.fc26.src.rpm                                                   
INFO: mock.py version 1.4.2 starting (python version = 3.6.2)...    
Start: init plugins               
INFO: selinux disabled            
Finish: init plugins              
Start: run                        
INFO: Start(/home/user/git/rpmbuild/SRPMS/psad-2.4.4-5.fc26.src.rpm)  Config(fedora-26-x86_64)                                         
Start: clean chroot               
Finish: clean chroot              
Start: chroot init                
INFO: calling preinit hooks       
INFO: enabled root cache          
INFO: enabled dnf cache           
Start: cleaning dnf metadata      
Finish: cleaning dnf metadata     
INFO: enabled HW Info plugin      
Mock Version: 1.4.2               
INFO: Mock Version: 1.4.2         
Start: dnf install                
Last metadata expiration check: 0:00:00 ago on Wed 02 Aug 2017 09:42:02 AM CEST.                                                        
failed calculating RPMDB checksum 
ERROR: Exception(/home/user/git/rpmbuild/SRPMS/psad-2.4.4-5.fc26.src.rpm) Config(fedora-26-x86_64) 0 minutes 3 seconds                 
INFO: Results and/or logs in: /var/lib/mock/fedora-26-x86_64/result 
ERROR: Command failed:            
 # /usr/bin/dnf --installroot /var/lib/mock/fedora-26-x86_64/root/ --releasever 26 --disableplugin=local --setopt=deltarpm=False install @buildsys-build
Last metadata expiration check: 0:00:00 ago on Wed 02 Aug 2017 09:42:02 AM CEST.                                                        
failed calculating RPMDB checksum 

root.log says:
INFO buildroot.py:345:  Mock Version: 1.4.2
DEBUG util.py:122:  touching file: /var/lib/mock/fedora-26-x86_64/root/etc/fstab
DEBUG util.py:122:  touching file: /var/lib/mock/fedora-26-x86_64/root/etc/yum/yum.conf
DEBUG util.py:122:  touching file: /var/lib/mock/fedora-26-x86_64/root/etc/dnf.conf
DEBUG util.py:122:  touching file: /var/lib/mock/fedora-26-x86_64/root/var/log/yum.log
DEBUG util.py:109:  ensuring that dir exists: /var/lib/mock/fedora-26-x86_64/root/var/cache/dnf/
DEBUG util.py:606:  child environment: None
DEBUG util.py:533:  Executing command: ['/bin/mount', '-n', '--bind', '/var/cache/mock/fedora-26-x86_64/dnf_cache/', '/var/lib/mock/fedora-26-x86_64/root/var/cache/dnf/'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'en_US.UTF-8'} and shell False
DEBUG util.py:588:  Child return code was: 0
DEBUG buildroot.py:140:  rootdir = /var/lib/mock/fedora-26-x86_64/root
DEBUG buildroot.py:141:  resultdir = /var/lib/mock/fedora-26-x86_64/result
DEBUG util.py:109:  ensuring that dir exists: /var/lib/mock/fedora-26-x86_64/root/etc/pki/mock
DEBUG util.py:112:  creating dir: /var/lib/mock/fedora-26-x86_64/root/etc/pki/mock
DEBUG util.py:109:  ensuring that dir exists: /var/lib/mock/fedora-26-x86_64/root/etc/dnf
DEBUG util.py:112:  creating dir: /var/lib/mock/fedora-26-x86_64/root/etc/dnf
DEBUG package_manager.py:107:  ['/usr/bin/dnf', '--installroot', '/var/lib/mock/fedora-26-x86_64/root/', '--releasever', '26', '--disableplugin=local', '--setopt=deltarpm=False', 'install', '@buildsys-build']
DEBUG util.py:606:  child environment: None
DEBUG util.py:533:  Executing command: ['/usr/bin/dnf', '--installroot', '/var/lib/mock/fedora-26-x86_64/root/', '--releasever', '26', '--disableplugin=local', '--setopt=deltarpm=False', 'install', '@buildsys-build'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'en_US.UTF-8', 'LC_MESSAGES': 'C.UTF-8'} and shell False
DEBUG util.py:450:  Last metadata expiration check: 0:00:00 ago on Wed 02 Aug 2017 09:42:02 AM CEST.
DEBUG util.py:450:  failed calculating RPMDB checksum
DEBUG util.py:588:  Child return code was: 1
DEBUG util.py:197:  kill orphans
DEBUG util.py:606:  child environment: None
DEBUG util.py:533:  Executing command: ['/bin/umount', '-n', '/var/lib/mock/fedora-26-x86_64/root/var/cache/dnf/'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'en_US.UTF-8'} and shell False
DEBUG util.py:588:  Child return code was: 0

Setting the config mentioned by @22 worked for me too.

*** Bug 1474125 has been marked as a duplicate of this bug. ***

*** Bug 1476000 has been marked as a duplicate of this bug. ***

While I can change all plugins in Mock, I do not feel it is correct. I would like to ask DNF to try to fix their debuginfo-install plugin first.

I create a patch that should solve the problem (https://github.com/rpm-software-management/dnf/pull/896). But the solution is in dnf.

dnf-plugins-extras-2.0.3-1.fc27 dnf-plugins-core-2.1.4-1.fc27 dnf-2.7.2-1.fc27 libdnf-0.10.1-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-faf235c683

dnf-plugins-extras-2.0.3-1.fc26 dnf-plugins-core-2.1.4-1.fc26 dnf-2.7.2-1.fc26 libdnf-0.10.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-70a8618065

dnf-2.7.2-1.fc26, dnf-plugins-core-2.1.4-1.fc26, dnf-plugins-extras-2.0.3-1.fc26, libdnf-0.10.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-70a8618065

dnf-2.7.2-1.fc27, dnf-plugins-core-2.1.4-1.fc27, dnf-plugins-extras-2.0.3-1.fc27, libdnf-0.10.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-faf235c683

dnf-2.7.2-1.fc27, dnf-plugins-core-2.1.4-1.fc27, dnf-plugins-extras-2.0.3-1.fc27, libdnf-0.10.1-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

dnf-2.7.2-1.fc26, dnf-plugins-core-2.1.4-1.fc26, dnf-plugins-extras-2.0.3-1.fc26, libdnf-0.10.1-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.