Bug 1205564 - systemd creates 'var/lib/machines' btrfs subvolumes in mock root
Summary: systemd creates 'var/lib/machines' btrfs subvolumes in mock root
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mock
Version: 21
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jan Synacek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1195082 (view as bug list)
Depends On:
Blocks: 1289471
TreeView+ depends on / blocked
 
Reported: 2015-03-25 08:37 UTC by Jan Synacek
Modified: 2015-12-08 09:03 UTC (History)
19 users (show)

Fixed In Version: mock-1.2.10-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1289471 (view as bug list)
Environment:
Last Closed: 2015-05-01 16:51:08 UTC
Type: Bug


Attachments (Terms of Use)
patch v1 (1.92 KB, patch)
2015-04-02 09:16 UTC, Jan Synacek
no flags Details | Diff
patch v2 (1.97 KB, patch)
2015-04-02 11:10 UTC, Jan Synacek
no flags Details | Diff
build.log on el7 (1.74 KB, text/plain)
2015-06-29 07:06 UTC, roidelapluie
no flags Details
state.log on el7 (455 bytes, text/plain)
2015-06-29 07:06 UTC, roidelapluie
no flags Details
root.log on el7 (29.00 KB, text/plain)
2015-06-29 07:08 UTC, roidelapluie
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1223467 0 unspecified CLOSED Suppress subvolume creation from tmpfiles when doing installroots 2021-02-22 00:41:40 UTC

Internal Links: 1223467

Description Jan Synacek 2015-03-25 08:37:07 UTC
Description of problem:
When using mock to build packages, systemd creates 'var/lib/machines' btrfs subvolume in every mock root that is created. The problem is that the mock then doesn't work, not even when run under root privileges, as the mock buildroot is always cleaned up using rm, which fails on the subvolume.


Version-Release number of selected component (if applicable):
systemd-216-21.fc21.x86_64


How reproducible:
Always


Steps to Reproduce:
1. Setup an F21 installation with a btrfs root
2. Install mock and try to build a package, for example for rawhide or F21
3. Observe the error:

~/work/git/fedora/mc (master)$ fedpkg mockbuild

Wrote: /home/jsynacek/work/git/fedora/mc/mc-4.8.14-1.fc23.src.rpm
INFO: mock.py version 1.2.7 starting (python version = 2.7.8)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
INFO: Start(/home/jsynacek/work/git/fedora/mc/mc-4.8.14-1.fc23.src.rpm)  Config(rawhide-x86_64)
Start: clean chroot
ERROR: Exception(/home/jsynacek/work/git/fedora/mc/mc-4.8.14-1.fc23.src.rpm) Config(rawhide-x86_64) 0 minutes 0 seconds
INFO: Results and/or logs in: /home/jsynacek/work/git/fedora/mc/results_mc/4.8.14/1.fc23
INFO: Cleaning up build root ('cleanup_on_failure=True')
Start: clean chroot

ERROR: [Errno 1] Operation not permitted: '/var/lib/mock/rawhide-x86_64/root/var/lib/machines'

ERROR: The most common cause for this error is trying to run /usr/sbin/mock as an unprivileged user.
ERROR: Check your path to make sure that /usr/bin/ is listed before /usr/sbin, or manually run /usr/bin/mock to see if that fixes this problem.


Actual results:
An error is thrown because there is a subvolume in the mock buildroot.


Expected results:
systemd doesn't create any subvolumes in mock roots, so no errors are shown when using mock.


Additional info:
$ sudo btrfs subv list / | grep mock
ID 481 gen 22844 top level 258 path var/lib/mock/rawhide-x86_64/root/var/lib/machines

$ sudo btrfs subv list / | grep 'ID 258'
ID 258 gen 22991 top level 5 path root

Comment 1 Jan Synacek 2015-03-25 08:41:54 UTC
*** Bug 1195082 has been marked as a duplicate of this bug. ***

Comment 2 Jan Synacek 2015-03-26 14:16:03 UTC
Also reproduced with systemd-219-8.fc23 in rawhide.

Comment 3 Jan Synacek 2015-04-01 12:24:42 UTC
The problem is that, in the mock's chroot, there is the standard /usr/lib/tmpfiles.d/var.conf file present which contains "v  /var/lib/machines 0700 - - -".

Patch is in the works.

Comment 4 Jan Synacek 2015-04-02 09:15:44 UTC
After a discussion with the maintainer of mock, he agreed that mock itself should be able to deal with subvolumes created inside it.

Comment 5 Jan Synacek 2015-04-02 09:16:36 UTC
Created attachment 1010082 [details]
patch v1

Comment 6 Jan Synacek 2015-04-02 11:10:52 UTC
Created attachment 1010109 [details]
patch v2

Fixed paths in the patch, so it applies correctly.

Comment 7 Fabrice Bellet 2015-04-03 12:41:14 UTC
I just tested the patch, and it works for me. Thanks!

Comment 8 Miroslav Suchý 2015-04-16 12:04:19 UTC
Patch applied as:
* 757f11b delete btrfs subvolumes on exit [RHBZ#1205564]

Thank you.

Comment 9 Miroslav Suchý 2015-04-16 13:22:15 UTC
I had to do one more commit:
* 4f274ce ignore btrfs errors on non-btrfs systems [RHBZ#1205564]

And ignore btrfs errors completely as it fails on non btrfs mounts with:
$ btrfs subv list /var/lib/mock
ERROR: can't perform the search - Inappropriate ioctl for device
ERROR: can't get rootid for '/var/lib/mock'

and return non-zero status code which cause mock to halt. If you know some way how to detect that if path contains btrfs and does not yield error, please let me know.

Comment 10 Jan Synacek 2015-04-20 05:53:29 UTC
Meh, I totally forgot to handle that case... I don't think there a better way to do this. It's probably safe to ignore the errors anyway, since if listing the subvolumes doesn't work, you can't know what to delete.

Thanks!

Comment 11 Fedora Update System 2015-04-29 13:16:51 UTC
mock-1.2.8-1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/mock-1.2.8-1.fc22

Comment 12 Fedora Update System 2015-04-29 13:17:16 UTC
mock-1.2.8-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/mock-1.2.8-1.fc21

Comment 13 Fedora Update System 2015-04-29 13:17:56 UTC
mock-1.2.8-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.2.8-1.fc20

Comment 14 Fedora Update System 2015-04-29 13:19:20 UTC
mock-1.2.8-1.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/mock-1.2.8-1.el7

Comment 15 Fedora Update System 2015-04-29 13:25:59 UTC
mock-1.2.8-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.2.8-1.el6

Comment 16 Fedora Update System 2015-04-29 19:19:03 UTC
Package mock-1.2.8-1.el7:
* should fix your issue,
* was pushed to the Fedora EPEL 7 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing mock-1.2.8-1.el7'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2015-6057/mock-1.2.8-1.el7
then log in and leave karma (feedback).

Comment 17 Fedora Update System 2015-05-01 16:51:08 UTC
mock-1.2.8-1.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2015-05-02 18:08:48 UTC
mock-1.2.8-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2015-05-12 20:46:33 UTC
mock-1.2.8-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2015-06-04 11:42:22 UTC
mock-1.2.10-1.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/mock-1.2.10-1.el7

Comment 21 Fedora Update System 2015-06-04 11:43:03 UTC
mock-1.2.10-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.2.10-1.el6

Comment 22 Fedora Update System 2015-06-20 21:10:46 UTC
mock-1.2.10-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 23 Fedora Update System 2015-06-20 21:13:39 UTC
mock-1.2.10-1.el7 has been pushed to the Fedora EPEL 7 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 24 roidelapluie 2015-06-29 07:06:09 UTC
Created attachment 1044196 [details]
build.log on el7

Comment 25 roidelapluie 2015-06-29 07:06:47 UTC
Created attachment 1044197 [details]
state.log on el7

Comment 26 roidelapluie 2015-06-29 07:08:49 UTC
Created attachment 1044198 [details]
root.log on el7

Comment 27 roidelapluie 2015-06-29 07:26:12 UTC
I still see this bug with mock-1.2.10-1.el7. See attached log files.

Comment 28 roidelapluie 2015-06-29 07:40:18 UTC
I am hit by another bug that I could find with mock --trace. Thanks.

Comment 29 roidelapluie 2015-06-29 07:42:23 UTC
(In reply to roidelapluie from comment #27)
> I still see this bug with mock-1.2.10-1.el7. See attached log files.

Do not take this into consideration. The actual error I have is error: File /builddir/build/SOURCES/d150e76ec83d9e6da5d98c908729434730bc3d1b.tar.gz: No such file or directory
but I do not see this error at any time in the mock output without --trace. I wac confused.

Comment 30 Ross Brattain 2015-06-30 21:33:53 UTC
mock-1.2.10-1.el7 still fails for me in a lxc centos 7 container

DEBUG: kill orphans
DEBUG: child environment: None
DEBUG: Executing command: ['btrfs', 'subv', 'list', '/home/mockbuild/mock/'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} and shell False
DEBUG: Child return code was: 1
Finish: clean chroot
ERROR: [Errno 1] Operation not permitted
ERROR: The most common cause for this error is trying to run /usr/sbin/mock as an unprivileged user.
ERROR: Check your path to make sure that /usr/bin/ is listed before /usr/sbin, or manually run /usr/bin/mock to see if that fixes this problem.


How do I disable mock using brtfs?

Comment 31 Ross Brattain 2015-06-30 21:58:52 UTC
(In reply to Ross Brattain from comment #30)
> mock-1.2.10-1.el7 still fails for me in a lxc centos 7 container

nevermind, btrfs isn't the problem.  it looks like it is failing to chroot in the lxc container.

Comment 32 secondary 2015-12-07 23:24:05 UTC
Doesn't work on my system.

First, some proof:

    $ dnf list installed mock
    Installed Packages
    mock.noarch     1.2.13-2.fc22   @updates


    $ mock --clean --trace # excerpt
    INFO: ENTER umount(<mockbuild.mounts.FileSystemMountPoint object at 0x7fc93078c320>)
    INFO: LEAVE umount --> None

    INFO: LEAVE umountall --> None

    INFO: ENTER call_hooks(<mockbuild.plugin.Plugins object at 0x7fc93c994b70>, 'umount_root')
    INFO: LEAVE call_hooks --> None

    INFO: ENTER _unlock_buildroot(<mockbuild.buildroot.Buildroot object at 0x7fc93077f9e8>)
    INFO: LEAVE _unlock_buildroot --> None

    INFO: EXCEPTION: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines'
    Traceback (most recent call last):
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/buildroot.py", line 563, in delete
        util.rmtree(self.basedir, selinux=self.selinux)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 145, in rmtree
        os.rmdir(path)
    PermissionError: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines'
    INFO: LEAVE delete --> EXCEPTION RAISED

    INFO: EXCEPTION: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines'
    Traceback (most recent call last):
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/backend.py", line 72, in clean
        self.buildroot.delete()
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/buildroot.py", line 563, in delete
        util.rmtree(self.basedir, selinux=self.selinux)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
        rmtree(fullname, selinux=selinux, exclude=exclude)
      File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
        result = func(*args, **kw)
      File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 145, in rmtree
        os.rmdir(path)
    PermissionError: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines'
    INFO: LEAVE clean --> EXCEPTION RAISED

    INFO: ENTER finalize(<mockbuild.buildroot.Buildroot object at 0x7fc93077f9e8>)
    INFO: ENTER make_chroot_path(<mockbuild.buildroot.Buildroot object at 0x7fc93077f9e8>)
    INFO: LEAVE make_chroot_path --> /var/lib/mock/fedora-22-x86_64/root

It doesn't work because of my btrfs layout:


    $ btrfs subvolume list / | egrep -i "mock|var|root"
    ID 257 gen 935842 top level 5 path root
    ID 258 gen 935847 top level 5 path var
    ID 724 gen 935172 top level 258 path var/lib/machines
    ID 3157 gen 935516 top level 258 path var/lib/mock/fedora-22-x86_64/root/var/lib/machines

    $ btrfs subvolume list /var/lib/mock/fedora-22-x86_64/ | egrep -i "mock|var|root"
    ID 257 gen 935851 top level 5 path root
    ID 258 gen 935857 top level 5 path var
    ID 3157 gen 935516 top level 258 path lib/mock/fedora-22-x86_64/root/var/lib/machines
                                          |
notice -----------------------------------|

This is because subvolumes are trees, which makes parsing more difficult ... but "btrfs subvolume list" does all the heavy lifting given the appropriate arguments.

    $ btrfs subvolume list -a /var/lib/mock/fedora-22-x86_64 | egrep -i "mock|var|root"
    ID 257 gen 935878 top level 5 path <FS_TREE>/root
    ID 258 gen 935880 top level 5 path <FS_TREE>/var
    ID 724 gen 935172 top level 258 path var/lib/machines
    ID 3157 gen 935516 top level 258 path var/lib/mock/fedora-22-x86_64/root/var/lib/machines

    $ btrfs subvolume list -o /var/lib/mock/fedora-22-x86_64
    ID 724 gen 935172 top level 258 path var/lib/machines
    ID 3157 gen 935516 top level 258 path var/lib/mock/fedora-22-x86_64/root/var/lib/machines

Proposed workaround: modify find_btrfs_in_chroot() to add the above "-o" flag:

def find_btrfs_in_chroot(mockdir, chroot_path):
    ...
    try:
       for l in do(["btrfs", "subv", "list", "-o", mockdir], returnOutput=1)[:-1].splitlines():
    ...

Comment 33 Miroslav Suchý 2015-12-08 09:03:18 UTC
Cloned as separate issue in bug 1289471


Note You need to log in before you can comment on or make changes to this bug.