Description of problem: When using mock to build packages, systemd creates 'var/lib/machines' btrfs subvolume in every mock root that is created. The problem is that the mock then doesn't work, not even when run under root privileges, as the mock buildroot is always cleaned up using rm, which fails on the subvolume. Version-Release number of selected component (if applicable): systemd-216-21.fc21.x86_64 How reproducible: Always Steps to Reproduce: 1. Setup an F21 installation with a btrfs root 2. Install mock and try to build a package, for example for rawhide or F21 3. Observe the error: ~/work/git/fedora/mc (master)$ fedpkg mockbuild Wrote: /home/jsynacek/work/git/fedora/mc/mc-4.8.14-1.fc23.src.rpm INFO: mock.py version 1.2.7 starting (python version = 2.7.8)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run INFO: Start(/home/jsynacek/work/git/fedora/mc/mc-4.8.14-1.fc23.src.rpm) Config(rawhide-x86_64) Start: clean chroot ERROR: Exception(/home/jsynacek/work/git/fedora/mc/mc-4.8.14-1.fc23.src.rpm) Config(rawhide-x86_64) 0 minutes 0 seconds INFO: Results and/or logs in: /home/jsynacek/work/git/fedora/mc/results_mc/4.8.14/1.fc23 INFO: Cleaning up build root ('cleanup_on_failure=True') Start: clean chroot ERROR: [Errno 1] Operation not permitted: '/var/lib/mock/rawhide-x86_64/root/var/lib/machines' ERROR: The most common cause for this error is trying to run /usr/sbin/mock as an unprivileged user. ERROR: Check your path to make sure that /usr/bin/ is listed before /usr/sbin, or manually run /usr/bin/mock to see if that fixes this problem. Actual results: An error is thrown because there is a subvolume in the mock buildroot. Expected results: systemd doesn't create any subvolumes in mock roots, so no errors are shown when using mock. Additional info: $ sudo btrfs subv list / | grep mock ID 481 gen 22844 top level 258 path var/lib/mock/rawhide-x86_64/root/var/lib/machines $ sudo btrfs subv list / | grep 'ID 258' ID 258 gen 22991 top level 5 path root
*** Bug 1195082 has been marked as a duplicate of this bug. ***
Also reproduced with systemd-219-8.fc23 in rawhide.
The problem is that, in the mock's chroot, there is the standard /usr/lib/tmpfiles.d/var.conf file present which contains "v /var/lib/machines 0700 - - -". Patch is in the works.
After a discussion with the maintainer of mock, he agreed that mock itself should be able to deal with subvolumes created inside it.
Created attachment 1010082 [details] patch v1
Created attachment 1010109 [details] patch v2 Fixed paths in the patch, so it applies correctly.
I just tested the patch, and it works for me. Thanks!
Patch applied as: * 757f11b delete btrfs subvolumes on exit [RHBZ#1205564] Thank you.
I had to do one more commit: * 4f274ce ignore btrfs errors on non-btrfs systems [RHBZ#1205564] And ignore btrfs errors completely as it fails on non btrfs mounts with: $ btrfs subv list /var/lib/mock ERROR: can't perform the search - Inappropriate ioctl for device ERROR: can't get rootid for '/var/lib/mock' and return non-zero status code which cause mock to halt. If you know some way how to detect that if path contains btrfs and does not yield error, please let me know.
Meh, I totally forgot to handle that case... I don't think there a better way to do this. It's probably safe to ignore the errors anyway, since if listing the subvolumes doesn't work, you can't know what to delete. Thanks!
mock-1.2.8-1.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/mock-1.2.8-1.fc22
mock-1.2.8-1.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/mock-1.2.8-1.fc21
mock-1.2.8-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/mock-1.2.8-1.fc20
mock-1.2.8-1.el7 has been submitted as an update for Fedora EPEL 7. https://admin.fedoraproject.org/updates/mock-1.2.8-1.el7
mock-1.2.8-1.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/mock-1.2.8-1.el6
Package mock-1.2.8-1.el7: * should fix your issue, * was pushed to the Fedora EPEL 7 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=epel-testing mock-1.2.8-1.el7' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-EPEL-2015-6057/mock-1.2.8-1.el7 then log in and leave karma (feedback).
mock-1.2.8-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.2.8-1.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.2.8-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.2.10-1.el7 has been submitted as an update for Fedora EPEL 7. https://admin.fedoraproject.org/updates/mock-1.2.10-1.el7
mock-1.2.10-1.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/mock-1.2.10-1.el6
mock-1.2.10-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.2.10-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
Created attachment 1044196 [details] build.log on el7
Created attachment 1044197 [details] state.log on el7
Created attachment 1044198 [details] root.log on el7
I still see this bug with mock-1.2.10-1.el7. See attached log files.
I am hit by another bug that I could find with mock --trace. Thanks.
(In reply to roidelapluie from comment #27) > I still see this bug with mock-1.2.10-1.el7. See attached log files. Do not take this into consideration. The actual error I have is error: File /builddir/build/SOURCES/d150e76ec83d9e6da5d98c908729434730bc3d1b.tar.gz: No such file or directory but I do not see this error at any time in the mock output without --trace. I wac confused.
mock-1.2.10-1.el7 still fails for me in a lxc centos 7 container DEBUG: kill orphans DEBUG: child environment: None DEBUG: Executing command: ['btrfs', 'subv', 'list', '/home/mockbuild/mock/'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} and shell False DEBUG: Child return code was: 1 Finish: clean chroot ERROR: [Errno 1] Operation not permitted ERROR: The most common cause for this error is trying to run /usr/sbin/mock as an unprivileged user. ERROR: Check your path to make sure that /usr/bin/ is listed before /usr/sbin, or manually run /usr/bin/mock to see if that fixes this problem. How do I disable mock using brtfs?
(In reply to Ross Brattain from comment #30) > mock-1.2.10-1.el7 still fails for me in a lxc centos 7 container nevermind, btrfs isn't the problem. it looks like it is failing to chroot in the lxc container.
Doesn't work on my system. First, some proof: $ dnf list installed mock Installed Packages mock.noarch 1.2.13-2.fc22 @updates $ mock --clean --trace # excerpt INFO: ENTER umount(<mockbuild.mounts.FileSystemMountPoint object at 0x7fc93078c320>) INFO: LEAVE umount --> None INFO: LEAVE umountall --> None INFO: ENTER call_hooks(<mockbuild.plugin.Plugins object at 0x7fc93c994b70>, 'umount_root') INFO: LEAVE call_hooks --> None INFO: ENTER _unlock_buildroot(<mockbuild.buildroot.Buildroot object at 0x7fc93077f9e8>) INFO: LEAVE _unlock_buildroot --> None INFO: EXCEPTION: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines' Traceback (most recent call last): File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/buildroot.py", line 563, in delete util.rmtree(self.basedir, selinux=self.selinux) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 145, in rmtree os.rmdir(path) PermissionError: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines' INFO: LEAVE delete --> EXCEPTION RAISED INFO: EXCEPTION: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines' Traceback (most recent call last): File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/backend.py", line 72, in clean self.buildroot.delete() File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/buildroot.py", line 563, in delete util.rmtree(self.basedir, selinux=self.selinux) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace result = func(*args, **kw) File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 145, in rmtree os.rmdir(path) PermissionError: [Errno 1] Operation not permitted: '/var/lib/mock/fedora-22-x86_64/root/var/lib/machines' INFO: LEAVE clean --> EXCEPTION RAISED INFO: ENTER finalize(<mockbuild.buildroot.Buildroot object at 0x7fc93077f9e8>) INFO: ENTER make_chroot_path(<mockbuild.buildroot.Buildroot object at 0x7fc93077f9e8>) INFO: LEAVE make_chroot_path --> /var/lib/mock/fedora-22-x86_64/root It doesn't work because of my btrfs layout: $ btrfs subvolume list / | egrep -i "mock|var|root" ID 257 gen 935842 top level 5 path root ID 258 gen 935847 top level 5 path var ID 724 gen 935172 top level 258 path var/lib/machines ID 3157 gen 935516 top level 258 path var/lib/mock/fedora-22-x86_64/root/var/lib/machines $ btrfs subvolume list /var/lib/mock/fedora-22-x86_64/ | egrep -i "mock|var|root" ID 257 gen 935851 top level 5 path root ID 258 gen 935857 top level 5 path var ID 3157 gen 935516 top level 258 path lib/mock/fedora-22-x86_64/root/var/lib/machines | notice -----------------------------------| This is because subvolumes are trees, which makes parsing more difficult ... but "btrfs subvolume list" does all the heavy lifting given the appropriate arguments. $ btrfs subvolume list -a /var/lib/mock/fedora-22-x86_64 | egrep -i "mock|var|root" ID 257 gen 935878 top level 5 path <FS_TREE>/root ID 258 gen 935880 top level 5 path <FS_TREE>/var ID 724 gen 935172 top level 258 path var/lib/machines ID 3157 gen 935516 top level 258 path var/lib/mock/fedora-22-x86_64/root/var/lib/machines $ btrfs subvolume list -o /var/lib/mock/fedora-22-x86_64 ID 724 gen 935172 top level 258 path var/lib/machines ID 3157 gen 935516 top level 258 path var/lib/mock/fedora-22-x86_64/root/var/lib/machines Proposed workaround: modify find_btrfs_in_chroot() to add the above "-o" flag: def find_btrfs_in_chroot(mockdir, chroot_path): ... try: for l in do(["btrfs", "subv", "list", "-o", mockdir], returnOutput=1)[:-1].splitlines(): ...
Cloned as separate issue in bug 1289471