Description of problem: Builds fail in private koji since updating to mock-1.4.2-1.el7.noarch (from mock-1.3.4-1.el7.noarch) Version-Release number of selected component (if applicable): mock-1.4.2-1.el7.noarch How reproducible: always Actual results: Here's the mock_output.log from our koji when building a package named rpm-build-tools: 2017-07-14 16:42:03,366 [INFO] {721} koji.build.buildroot:428 Running mock as kojibuilder INFO: mock.py version 1.4.2 starting (python version = 2.7.5)... Start: init plugins INFO: selinux disabled Finish: init plugins Start: run Start: clean chroot Finish: clean chroot Start: chroot init INFO: calling preinit hooks INFO: enabled HW Info plugin Mock Version: 1.4.2 INFO: Mock Version: 1.4.2 Start: yum install Finish: yum install Finish: chroot init INFO: Installed packages: Finish: run 2017-07-14 16:43:55,031 [INFO] {1480} koji.build.buildroot:428 Running mock as kojibuilder INFO: mock.py version 1.4.2 starting (python version = 2.7.5)... Start: init plugins INFO: selinux disabled Finish: init plugins Start: run Start: chroot init INFO: calling preinit hooks INFO: enabled HW Info plugin Mock Version: 1.4.2 INFO: Mock Version: 1.4.2 Finish: chroot init INFO: Running in chroot: ['make', 'sources'] Start: chroot ['make', 'sources'] Finish: chroot ['make', 'sources'] Traceback (most recent call last): File "/usr/libexec/mock/mock", line 934, in <module> main() File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 737, in main run_command(options, args, config_opts, commands, buildroot, state) File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 785, in run_command commands.chroot(args, options) File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 335, in chroot nspawn_args=self.config['nspawn_args']) File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 202, in doChroot env=env, shell=shell, *args, **kargs) File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 543, in do preexec_fn=preexec, File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory: '/tmp/scmroot/rpm-build-tools' Additional info: If I downgrade back to mock-1.3.4-1.el7.noarch then all is well again.
I think I have this same bug on Fedora 25 and private Koji. Happens every time for me also. mock_output.log: 2017-07-28 10:41:24,748 [INFO] {18220} koji.build.buildroot:440 Running mock as kojiadmin INFO: mock.py version 1.4.2 starting (python version = 3.5.3)... Start: init plugins INFO: selinux disabled Finish: init plugins Start: run Start: clean chroot Finish: clean chroot Start: chroot init INFO: calling preinit hooks INFO: enabled HW Info plugin Mock Version: 1.4.2 INFO: Mock Version: 1.4.2 Start: dnf install Finish: dnf install Finish: chroot init INFO: Installed packages: Finish: run 2017-07-28 10:43:49,856 [INFO] {18741} koji.build.buildroot:440 Running mock as kojiadmin INFO: mock.py version 1.4.2 starting (python version = 3.5.3)... Start: init plugins INFO: selinux disabled Finish: init plugins Start: run Start: chroot init INFO: calling preinit hooks INFO: enabled HW Info plugin Mock Version: 1.4.2 INFO: Mock Version: 1.4.2 Finish: chroot init INFO: Running in chroot: ['fedpkg', 'sources'] Start: chroot ['fedpkg', 'sources'] Finish: chroot ['fedpkg', 'sources'] ERROR: Exception occurred in preexec_fn. Traceback (most recent call last): File "/usr/libexec/mock/mock", line 934, in <module> main() File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 737, in main run_command(options, args, config_opts, commands, buildroot, state) File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 785, in run_command commands.chroot(args, options) File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.5/site-packages/mockbuild/backend.py", line 335, in chroot nspawn_args=self.config['nspawn_args']) File "/usr/lib/python3.5/site-packages/mockbuild/buildroot.py", line 202, in doChroot env=env, shell=shell, *args, **kargs) File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.5/site-packages/mockbuild/util.py", line 542, in do preexec_fn=preexec, File "/usr/lib64/python3.5/subprocess.py", line 676, in __init__ restore_signals, start_new_session) File "/usr/lib64/python3.5/subprocess.py", line 1283, in _execute_child raise child_exception_type(err_msg) subprocess.SubprocessError: Exception occurred in preexec_fn. I can workaround this problem by setting new_chroot to false (koji edit-tag <build-tag> -x mock.new_chroot=False). This error happened with both 1.4.2-1.fc25 and 1.3.4-1.fc25 if new_chroot is true. I think the problem here is related to systemd-nspawn mounting tmpfs on /tmp in the container. I have manually systemd-nspawn'd into a buildroot left by this failed build and checked that /tmp/scmroot wasn't visible in the filesystem. After running 'umount /tmp' /tmp/scmroot and its subfolders become visible in the container.
Riku, I had just come to the same conclusion last night testing my https://pagure.io/koji/pull-request/419 on Fedora 26 where I met this problem again. Your timing was impeccable as I was totally unaware that mock could be configured via koji tags in the way you show above. I've always relied on site-defaults.cfg which never felt right -- too loosely coupled. Thank you for sharing both your workaround and details of your investigation! I can also confirm your workaround resolves the issue for me.
How do you mount that /tmp/scmroot/ ? With nspawn /tmp is mounted by nspawn itself. I already moved mounting of user defined mounts here and there and there is no perfect solution for that.
(In reply to Miroslav Suchý from comment #3) > How do you mount that /tmp/scmroot/ ? I don't do anything with that directly. The koji builders do however. It looks like there a bug report there, now: https://pagure.io/koji/issue/531
Ah I see: msuchy@dri/~/projects/koji{master}]$ git grep scmroot builder/kojid: scmdir = buildroot.rootdir() + '/tmp/scmroot' builder/kojid: scmdir = broot.rootdir() + '/tmp/scmroot' It seems that Koji will write there some data before running mock. This is bad behaviour. I really cannot do anything about it. Container will simply mount tmpfs over /tmp which will mask previous content. You can either write it to some other directory. Or even better write it to host /tmp and mount it inside chroot using bind mount plugin which should be mounted after mounting system volumes: https://github.com/rpm-software-management/mock/wiki/Plugin-BindMount While I understand that it is change in behaviour I think this should be fixed on the Koji side. I will close this bug, but I will be happy to assist you if you need some help with this issue.