Bug 1128036 - mock fails when kernel does not support NAMESPACE
Summary: mock fails when kernel does not support NAMESPACE
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: mock
Version: el6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miroslav Suchý
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-08 07:04 UTC by Robin Lee
Modified: 2016-12-30 20:07 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-30 20:07:22 UTC


Attachments (Terms of Use)

Description Robin Lee 2014-08-08 07:04:46 UTC
Description of problem:
Runnig a mock --rebuild, the error comes out at once.

The file really exists:

$ wc -l /proc/mounts 
9 /proc/mounts


calltrace:

ERROR: [Errno 2] No such file or directory: '/proc/mounts'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 694, in <module>
    main(retParams)
  File "/usr/sbin/mock", line 628, in main
    do_rebuild(config_opts, chroot, args)
  File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at 0x027D9A28>", line 3, in do_rebuild
    def do_rebuild(config_opts, chroot, srpms): return __decorated(config_opts, chroot, srpms)
  File "/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 276, in do_rebuild
    chroot.clean()
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend.clean at 0x027CD398>", line 3, in clean
    def clean(self): return __decorated(self)
  File "/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line 205, in clean
    self._umountall(nowarn=True)
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend._umountall at 0x027D8578>", line 3, in _umountall
    def _umountall(self, nowarn): return __decorated(self, nowarn)
  File "/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line 961, in _umountall
    mountpoints = open("/proc/mounts").read().strip().split("\n")
IOError: [Errno 2] No such file or directory: '/proc/mounts'

Version-Release number of selected component (if applicable):
mock-1.1.41-1.el6

Comment 1 Miroslav Suchý 2014-08-08 09:51:17 UTC
Can you please provide exact command line you run?
And the config file of chroot which you used?

Comment 2 Robin Lee 2014-08-10 05:27:40 UTC
We use a costumed config and a costumed distro based on CentOS 6.2 and kernel 2.6.32.

The full command is:
$ mock -r XXX-1-x86_64 --rebuild /home/ruibinli/rpmbuild/SRPMS/libvirt-1.2.6-1.zzz.src.rpm

The XXX-1-x86_64.cfg is like this (sensitive string replaced):

config_opts['root'] = 'XXX-1-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['legal_host_arches'] = ('x86_64',)
#config_opts['chroot_setup_cmd'] = 'groupinstall buildsys-build'
config_opts['useradd'] = '/usr/sbin/useradd -m -u %(uid)s -g %(gid)s -d /builddir %(user)s'
config_opts['chroot_setup_cmd'] = 'install bash bzip2 coreutils cpio diffutils XXX-release findutils gawk gcc gcc-c++ grep gzip info make patch rpm-build sed shadow-utils tar unzip util-linux-ng which bash curl gnupg make redhat-rpm-config'
# change release tag
config_opts['dist'] = 'zzz'  # only useful for --resultdir variable subst
config_opts['macros']['%vendor'] = "YYY"
#config_opts['macros']['%_smp_mflags'] = "-j20"
config_opts['macros']['%distribution'] = "XXX"
config_opts['macros']['%packager'] = "XXX team <t_os@YYY.com>"

#Speedup mock
#config_opts['plugin_conf']['root_cache_opts']['compress_program'] = "pigz"
#config_opts['plugin_conf']['tmpfs_enable'] = True
#config_opts['plugin_conf']['tmpfs_opts']['required_ram_mb'] = 2048
# config_opts['plugin_conf']['tmpfs_opts']['max_fs_size'] = '512m'

config_opts['plugin_conf']['ccache_enable'] = False
 

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=1
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1
syslog_ident=mock
syslog_device=

# repos
[base]
name=BaseOS
enabled=1
#mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os
mirrorlist=http://mirrorlist-XXX.YYY.com/?release=6.2&arch=x86_64&repo=os
failovermethod=priority

[updates]
name=updates
enabled=1
#mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=updates
mirrorlist=http://mirrorlist-XXX.YYY.com/?release=6.2&arch=x86_64&repo=updates
failovermethod=priority

[XXX]
name=XXX - tlinux 
mirrorlist=http://mirrorlist-XXX.YYY.com/?release=6.2&arch=x86_64&repo=tlinux
enabled=1

[XXX-testing]
name=XXX-testing - tlinux-testing
mirrorlist=http://mirrorlist-XXX.YYY.com/?release=6.2&arch=x86_64&repo=tlinux-testing
enabled=1

[epel]
name=epel
enabled=1
#mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-6&arch=x86_64
mirrorlist=http://mirrorlist-XXX.YYY.com/?release=6.2&arch=x86_64&repo=epel
failovermethod=priority
gpgcheck=0

[testing]
name=epel-testing
enabled=0
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=testing-epel6&arch=x86_64
failovermethod=priority

[local]
name=local
baseurl=http://kojipkgs.fedoraproject.org/repos/dist-6E-epel-build/latest/x86_64/
cost=2000
enabled=0
"""

Comment 3 Scott K Logan 2014-09-15 07:20:49 UTC
I'm hitting this same thing on my F20 ARM build slaves. I was using a custom .cfg, but it does the same thing with the vanilla ones. Also tried falling back to mock 1.1.35-1, but acts the same as current (1.1.41-1).

Not showing any problems on F19 ARM build slaves (mock 1.1.38-1).

Comment 4 Rob Gilton 2014-10-05 20:05:52 UTC
I'm also seeing this on an F20 ARM device.  Placing a subprocess.call(["ls","/proc"]) just before the line that attempts to read /proc/mounts shows that there is nothing in the /proc directory at this point.

Comment 5 Michael Simacek 2014-10-06 13:38:23 UTC
If /proc/mounts exists both before and after mock execution, it seems that it didn't correctly propagate into mock's namespace. Can you post output of
findmnt -o SOURCE,TARGET,PROPAGATION

Other option is that it was accidentally unmounted, but the change didn't propagate back. Can you post the buildroot path - output of
mock --print-root-path

It seems that it may misbehave when the path contains spaces or some other special characters.

Comment 6 Vicente Olivert Riera 2014-11-05 12:39:17 UTC
Hello, I'm having the same problem here in the MIPS port of Fedora 20 I'm trying to do.

I'm running a small Fedora 20 in a MIPS board, but the / partition is mounted through NFS.

(In reply to Michael Simacek from comment #5)
> If /proc/mounts exists both before and after mock execution, it seems that
> it didn't correctly propagate into mock's namespace. Can you post output of
> findmnt -o SOURCE,TARGET,PROPAGATION

-bash-4.2$ findmnt -o SOURCE,TARGET,PROPAGATION
SOURCE                              TARGET                           PROPAGATION
192.168.154.104:/home/vincent/nfs/fedora/
                                    /                                shared
proc                                |-/proc                          shared
sysfs                               |-/sys                           shared
tmpfs                               | |-/sys/fs/cgroup               shared
cgroup                              | | |-/sys/fs/cgroup/systemd     shared
cgroup                              | | |-/sys/fs/cgroup/cpuset      shared
cgroup                              | | |-/sys/fs/cgroup/cpu,cpuacct shared
cgroup                              | | |-/sys/fs/cgroup/memory      shared
cgroup                              | | |-/sys/fs/cgroup/devices     shared
cgroup                              | | |-/sys/fs/cgroup/freezer     shared
cgroup                              | | `-/sys/fs/cgroup/blkio       shared
debugfs                             | |-/sys/kernel/debug            shared
fusectl                             | `-/sys/fs/fuse/connections     shared
devtmpfs                            |-/dev                           shared
tmpfs                               | |-/dev/shm                     shared
devpts                              | `-/dev/pts                     shared
tmpfs                               |-/run                           shared
tmpfs                               `-/tmp                           shared

> Other option is that it was accidentally unmounted, but the change didn't
> propagate back. Can you post the buildroot path - output of
> mock --print-root-path

-bash-4.2$ mock -r fedora-20-mipsel --print-root-path
/var/lib/mock/fedora-20-mipsel/root/

Some more information about the permissions of those directories:

-bash-4.2$ ls -l /var/lib/mock/fedora-20-mipsel/     
total 4
-rw-rw-r-- 1 root mock    0 Nov  5  2014 buildroot.lock
drwxr-sr-x 2 root mock 4096 Nov  5  2014 root

-bash-4.2$ ls -l /var/lib/mock/                 
total 4
drwxr-sr-x 3 root mock 4096 Nov  5  2014 fedora-20-mipsel

Comment 7 Miroslav Suchý 2014-11-18 16:43:44 UTC
Can you please try if this happen to you with
  --new-chroot
option? It is available since mock-1.2.0 (currently in updates-testing).

Comment 8 Vicente Olivert Riera 2014-11-18 17:35:47 UTC
(In reply to Miroslav Suchý from comment #7)
> Can you please try if this happen to you with
>   --new-chroot
> option? It is available since mock-1.2.0 (currently in updates-testing).

Thank you very much for your reply and for trying to help me. I have built the mock-1.2.0 rpm and installed it. And then I have tried to use it with the option you suggested.

-bash-4.2$ mock --version
1.2.0

-bash-4.2$ mock -r fedora-20-mipsel --rebuild --new-chroot bash-4.2.45-4.fc20.src.rpm
INFO: mock.py version 1.2.0 starting (python version = 2.7.5)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
warning: Failed to read auxiliary vector, /proc not mounted?
warning: Failed to read auxiliary vector, /proc not mounted?
warning: Failed to read auxiliary vector, /proc not mounted?
warning: Failed to read auxiliary vector, /proc not mounted?
warning: Failed to read auxiliary vector, /proc not mounted?
warning: Failed to read auxiliary vector, /proc not mounted?
warning: Failed to read auxiliary vector, /proc not mounted?
warning: Failed to read auxiliary vector, /proc not mounted?
INFO: Start(bash-4.2.45-4.fc20.src.rpm)  Config(fedora-20-mipsel)
Start: clean chroot
ERROR: Exception(bash-4.2.45-4.fc20.src.rpm) Config(fedora-20-mipsel) 0 minutes 0 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-20-mipsel/result
ERROR: [Errno 2] No such file or directory: '/proc/mounts'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 809, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 649, in main
    buildroot.finalize()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 475, in finalize
    self._umount_all()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 505, in _umount_all
    self._umount_residual()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 537, in _umount_residual
    current_mounts = get_our_mounts()
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 525, in get_our_mounts
    mountpoints = open("/proc/mounts").read().strip().split("\n")
IOError: [Errno 2] No such file or directory: '/proc/mounts'

Comment 9 Miroslav Suchý 2014-11-20 14:31:11 UTC
Just to confirm:
  mock -r fedora-20-mipsel --init
  sudo systemd-nspawn -D /var/lib/mock/fedora-20-mipsel/root
and then in that container
  ls /proc/mounts
will say: No such file or directory?

Can you please test if either one of these exists?
  /etc/mtab
  /proc/self/mounts

Comment 10 Vicente Olivert Riera 2014-11-20 14:58:37 UTC
-bash-4.2$ mock -r fedora-20-mipsel --init
INFO: mock.py version 1.2.0 starting (python version = 2.7.5)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: clean chroot
ERROR: [Errno 2] No such file or directory: '/proc/mounts'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 809, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 649, in main
    buildroot.finalize()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 475, in finalize
    self._umount_all()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 505, in _umount_all
    self._umount_residual()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 537, in _umount_residual
    current_mounts = get_our_mounts()
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 525, in get_our_mounts
    mountpoints = open("/proc/mounts").read().strip().split("\n")
IOError: [Errno 2] No such file or directory: '/proc/mounts'



-bash-4.2# systemd-nspawn -D /var/lib/mock/fedora-20-mipsel/root
Directory /var/lib/mock/fedora-20-mipsel/root doesn't look like an OS root directory (/etc/os-release is missing). Refusing.

-bash-4.2# cat /etc/os-release
NAME=Generic
VERSION="20 (Generic)"
ID=generic
VERSION_ID=20
PRETTY_NAME="Generic 20 (Generic)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:generic:generic:20"



-bash-4.2# ls /etc/mtab -l
lrwxrwxrwx 1 root root 12 Sep 29 16:55 /etc/mtab -> /proc/mounts

-bash-4.2# ls /proc/mounts -l
lrwxrwxrwx 1 root root 11 Nov 20 14:57 /proc/mounts -> self/mounts

-bash-4.2# ls /proc/self/mounts -l
-r--r--r-- 1 root root 0 Nov 20 14:57 /proc/self/mounts

Comment 11 Miroslav Suchý 2014-11-20 15:54:20 UTC
not even --init... hmmm. So once again:

  /usr/bin/yum --installroot /var/lib/mock/fedora-20-mipsel/root/ --releasever 20 install @buildsys-build
  sudo systemd-nspawn -D /var/lib/mock/fedora-20-mipsel/root
and then in that container
  ls /proc/mounts /etc/mtab /proc/self/mounts

BTW I today tried armhpf and this does not happen there.

Comment 12 Vicente Olivert Riera 2014-11-20 15:58:15 UTC
Well, I think that will not work. Are you trying to install a group of packages into that directory using yum? yum will try to fetch those packages from a repository, and there is no repository for Fedora 20 MIPS.

Anyway...

-bash-4.2# /usr/bin/yum --installroot /var/lib/mock/fedora-20-mipsel/root/ --releasever 20 install @buildsys-build
Plugin "puppetverify" can't be imported
Plugin "refresh-updatesd" can't be imported
Loaded plugins: aliases, auto-update-debuginfo, changelog, etckeeper,
              : fastestmirror, filter-data, fs-snapshot, keys, langpacks, list-
              : data, local, merge-conf, post-transaction-actions, priorities,
              : protectbase, ps, remove-with-leaves, rpm-warm-cache, security,
              : show-leaves, tmprepo, tsflags, upgrade-helper, verify,
              : versionlock
Config error: Command "updateinfo" already defined
-bash-4.2#

Comment 13 Miroslav Suchý 2014-11-20 16:14:02 UTC
yes. this is what mock is doing (althoug he will use yum.conf from mock.config), but usually if your arch is the same, this shortcut will work.
This yum command will prepare files for chroot.
And then systemd-nspawn will run container in that directory. And it should do all the hard work (mount /dev/shm, /proc etc.).
So I'm trying to find what is inside of your containers.

If none of
  /proc/mounts /etc/mtab /proc/self/mounts
exists, then it is fault of systemd.
If one of them exists, I can tune up mock code to use that one, if original does not exist.

Comment 14 Vicente Olivert Riera 2014-11-21 11:07:14 UTC
The /var/lib/mock/fedora-20-mipsel/root/ directory is empty.

Comment 15 Scott K Logan 2014-12-22 22:28:01 UTC
This bug is stopping builds on all of the versions of mock available in Fedora 21 on ARM as well.

I'm using a custom kernel of version 3.4.104 if that makes any difference...

Thanks,

--scott

Comment 16 Scott K Logan 2015-01-02 09:55:58 UTC
It took me some time, but I solved this issue on my ARM machines.

The root cause is that the kernel is missing CONFIG_NAMESPACES. This makes sense given that the platforms having trouble use different kernel configurations than x86 machines.

I think this is supposed to be checked for, but the test doesn't seem to work correctly. Is it a regression caused by [1]?

In any case, I enabled CONFIG_NAMESPACES and CONFIG_UTS_NS in my kernel, and mock works fine again.

Thanks,

--scott

[1] https://git.fedorahosted.org/cgit/mock.git/commit/?id=c08f5036d0b965289b5fd487380ace84f5a63f4f

Comment 17 Miroslav Suchý 2015-04-23 14:15:07 UTC
Notes for myself.
It seems that proper solution would be to warn user if their kernel does not support namespace.
Either 
 grep NAMESPACE /boot/config-$(uname -r)
or
 check return code on unshare (in util.py)

But since I do not have such kernel and it would take me some time, I am putting this on back burner.

Comment 18 Miroslav Suchý 2016-12-30 20:07:22 UTC
This was open for very long time. I still have no idea how to fix it. And it is so rare, that probably no one hit it.

For anyone who google this - just recompile your kernel and enable NAMESPACE - and mock will run without problem.

If anyone has nice idea how to detect it and print nice error with hint for user - please reopen this and tell me.


Note You need to log in before you can comment on or make changes to this bug.