Bug 1471282

I think I have this same bug on Fedora 25 and private Koji. Happens every time for me also. mock_output.log:


2017-07-28 10:41:24,748 [INFO] {18220} koji.build.buildroot:440 Running mock as kojiadmin
INFO: mock.py version 1.4.2 starting (python version = 3.5.3)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: clean chroot
Finish: clean chroot
Start: chroot init
INFO: calling preinit hooks
INFO: enabled HW Info plugin
Mock Version: 1.4.2
INFO: Mock Version: 1.4.2
Start: dnf install
Finish: dnf install
Finish: chroot init
INFO: Installed packages:
Finish: run
2017-07-28 10:43:49,856 [INFO] {18741} koji.build.buildroot:440 Running mock as kojiadmin
INFO: mock.py version 1.4.2 starting (python version = 3.5.3)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: chroot init
INFO: calling preinit hooks
INFO: enabled HW Info plugin
Mock Version: 1.4.2
INFO: Mock Version: 1.4.2
Finish: chroot init
INFO: Running in chroot: ['fedpkg', 'sources']
Start: chroot ['fedpkg', 'sources']
Finish: chroot ['fedpkg', 'sources']
ERROR: Exception occurred in preexec_fn.
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 934, in <module>
    main()
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 737, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 785, in run_command
    commands.chroot(args, options)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/backend.py", line 335, in chroot
    nspawn_args=self.config['nspawn_args'])
  File "/usr/lib/python3.5/site-packages/mockbuild/buildroot.py", line 202, in doChroot
    env=env, shell=shell, *args, **kargs)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/util.py", line 542, in do
    preexec_fn=preexec,
  File "/usr/lib64/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.5/subprocess.py", line 1283, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.


I can workaround this problem by setting new_chroot to false (koji edit-tag <build-tag> -x mock.new_chroot=False). This error happened with both 1.4.2-1.fc25 and 1.3.4-1.fc25 if new_chroot is true. I think the problem here is related to systemd-nspawn mounting tmpfs on /tmp in the container. I have manually systemd-nspawn'd into a buildroot left by this failed build and checked that /tmp/scmroot wasn't visible in the filesystem. After running 'umount /tmp' /tmp/scmroot and its subfolders become visible in the container.

Riku, I had just come to the same conclusion last night testing my https://pagure.io/koji/pull-request/419 on Fedora 26 where I met this problem again.  Your timing was impeccable as I was totally unaware that mock could be configured via koji tags in the way you show above.  I've always relied on site-defaults.cfg which never felt right -- too loosely coupled.  Thank you for sharing both your workaround and details of your investigation!

I can also confirm your workaround resolves the issue for me.

How do you mount that /tmp/scmroot/ ?

With nspawn /tmp is mounted by nspawn itself. I already moved mounting of user defined mounts here and there and there is no perfect solution for that.

(In reply to Miroslav Suchý from comment #3)
> How do you mount that /tmp/scmroot/ ?

I don't do anything with that directly.  The koji builders do however.

It looks like there a bug report there, now: 
https://pagure.io/koji/issue/531

Ah I see:
msuchy@dri/~/projects/koji{master}]$ git grep scmroot
builder/kojid:        scmdir = buildroot.rootdir() + '/tmp/scmroot'
builder/kojid:        scmdir = broot.rootdir() + '/tmp/scmroot'

It seems that Koji will write there some data before running mock. This is bad behaviour.
I really cannot do anything about it. Container will simply mount tmpfs over /tmp which will mask previous content.
You can either write it to some other directory. Or even better write it to host /tmp and mount it inside chroot using bind mount plugin which should be mounted after mounting system volumes:
https://github.com/rpm-software-management/mock/wiki/Plugin-BindMount

While I understand that it is change in behaviour I think this should be fixed on the Koji side.
I will close this bug, but I will be happy to assist you if you need some help with this issue.