Bug 1471282 - OSError: [Errno 2] No such file or directory: '/tmp/scmroot/$PACKAGE_NAME'
OSError: [Errno 2] No such file or directory: '/tmp/scmroot/$PACKAGE_NAME'
Status: CLOSED NOTABUG
Product: Fedora EPEL
Classification: Fedora
Component: mock (Show other bugs)
epel7
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Clark Williams
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-14 17:26 EDT by John Florian
Modified: 2017-09-18 14:39 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-09 04:26:07 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description John Florian 2017-07-14 17:26:19 EDT
Description of problem:
Builds fail in private koji since updating to mock-1.4.2-1.el7.noarch (from mock-1.3.4-1.el7.noarch)

Version-Release number of selected component (if applicable):
mock-1.4.2-1.el7.noarch

How reproducible:
always

Actual results:
Here's the mock_output.log from our koji when building a package named rpm-build-tools:

2017-07-14 16:42:03,366 [INFO] {721} koji.build.buildroot:428 Running mock as kojibuilder
INFO: mock.py version 1.4.2 starting (python version = 2.7.5)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: clean chroot
Finish: clean chroot
Start: chroot init
INFO: calling preinit hooks
INFO: enabled HW Info plugin
Mock Version: 1.4.2
INFO: Mock Version: 1.4.2
Start: yum install
Finish: yum install
Finish: chroot init
INFO: Installed packages:
Finish: run
2017-07-14 16:43:55,031 [INFO] {1480} koji.build.buildroot:428 Running mock as kojibuilder
INFO: mock.py version 1.4.2 starting (python version = 2.7.5)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: chroot init
INFO: calling preinit hooks
INFO: enabled HW Info plugin
Mock Version: 1.4.2
INFO: Mock Version: 1.4.2
Finish: chroot init
INFO: Running in chroot: ['make', 'sources']
Start: chroot ['make', 'sources']
Finish: chroot ['make', 'sources']
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 934, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 737, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 785, in run_command
    commands.chroot(args, options)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 335, in chroot
    nspawn_args=self.config['nspawn_args'])
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 202, in doChroot
    env=env, shell=shell, *args, **kargs)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 543, in do
    preexec_fn=preexec,
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory: '/tmp/scmroot/rpm-build-tools'



Additional info:
If I downgrade back to mock-1.3.4-1.el7.noarch then all is well again.
Comment 1 Riku Ahonen 2017-07-28 03:56:45 EDT
I think I have this same bug on Fedora 25 and private Koji. Happens every time for me also. mock_output.log:


2017-07-28 10:41:24,748 [INFO] {18220} koji.build.buildroot:440 Running mock as kojiadmin
INFO: mock.py version 1.4.2 starting (python version = 3.5.3)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: clean chroot
Finish: clean chroot
Start: chroot init
INFO: calling preinit hooks
INFO: enabled HW Info plugin
Mock Version: 1.4.2
INFO: Mock Version: 1.4.2
Start: dnf install
Finish: dnf install
Finish: chroot init
INFO: Installed packages:
Finish: run
2017-07-28 10:43:49,856 [INFO] {18741} koji.build.buildroot:440 Running mock as kojiadmin
INFO: mock.py version 1.4.2 starting (python version = 3.5.3)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: chroot init
INFO: calling preinit hooks
INFO: enabled HW Info plugin
Mock Version: 1.4.2
INFO: Mock Version: 1.4.2
Finish: chroot init
INFO: Running in chroot: ['fedpkg', 'sources']
Start: chroot ['fedpkg', 'sources']
Finish: chroot ['fedpkg', 'sources']
ERROR: Exception occurred in preexec_fn.
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 934, in <module>
    main()
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 737, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 785, in run_command
    commands.chroot(args, options)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/backend.py", line 335, in chroot
    nspawn_args=self.config['nspawn_args'])
  File "/usr/lib/python3.5/site-packages/mockbuild/buildroot.py", line 202, in doChroot
    env=env, shell=shell, *args, **kargs)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/util.py", line 542, in do
    preexec_fn=preexec,
  File "/usr/lib64/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.5/subprocess.py", line 1283, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.


I can workaround this problem by setting new_chroot to false (koji edit-tag <build-tag> -x mock.new_chroot=False). This error happened with both 1.4.2-1.fc25 and 1.3.4-1.fc25 if new_chroot is true. I think the problem here is related to systemd-nspawn mounting tmpfs on /tmp in the container. I have manually systemd-nspawn'd into a buildroot left by this failed build and checked that /tmp/scmroot wasn't visible in the filesystem. After running 'umount /tmp' /tmp/scmroot and its subfolders become visible in the container.
Comment 2 John Florian 2017-07-28 09:30:14 EDT
Riku, I had just come to the same conclusion last night testing my https://pagure.io/koji/pull-request/419 on Fedora 26 where I met this problem again.  Your timing was impeccable as I was totally unaware that mock could be configured via koji tags in the way you show above.  I've always relied on site-defaults.cfg which never felt right -- too loosely coupled.  Thank you for sharing both your workaround and details of your investigation!

I can also confirm your workaround resolves the issue for me.
Comment 3 Miroslav Suchý 2017-08-06 18:22:36 EDT
How do you mount that /tmp/scmroot/ ?

With nspawn /tmp is mounted by nspawn itself. I already moved mounting of user defined mounts here and there and there is no perfect solution for that.
Comment 4 John Florian 2017-08-08 09:32:07 EDT
(In reply to Miroslav Suchý from comment #3)
> How do you mount that /tmp/scmroot/ ?

I don't do anything with that directly.  The koji builders do however.

It looks like there a bug report there, now: 
https://pagure.io/koji/issue/531
Comment 5 Miroslav Suchý 2017-08-09 04:26:07 EDT
Ah I see:
msuchy@dri/~/projects/koji{master}]$ git grep scmroot
builder/kojid:        scmdir = buildroot.rootdir() + '/tmp/scmroot'
builder/kojid:        scmdir = broot.rootdir() + '/tmp/scmroot'

It seems that Koji will write there some data before running mock. This is bad behaviour.
I really cannot do anything about it. Container will simply mount tmpfs over /tmp which will mask previous content.
You can either write it to some other directory. Or even better write it to host /tmp and mount it inside chroot using bind mount plugin which should be mounted after mounting system volumes:
https://github.com/rpm-software-management/mock/wiki/Plugin-BindMount

While I understand that it is change in behaviour I think this should be fixed on the Koji side.
I will close this bug, but I will be happy to assist you if you need some help with this issue.

Note You need to log in before you can comment on or make changes to this bug.