Description of problem: https://copr.fedorainfracloud.org/coprs/vbatts/envoy/builds/ Attempting to build in copr, and while the build of my RPM is successful, the cleanup of the chroot fails, resulting in a failed build. I had opened https://bugzilla.redhat.com/show_bug.cgi?id=1493244, but was instructed to open this bug with mock as well. end of that log shows a successful build, but failed cleanup: ``` + cd /builddir/build/BUILD + cd envoy-ec3446e9b3631568db6223e9eaf37fc6ce9f6cf2 + /usr/bin/rm -rf /builddir/build/BUILDROOT/envoy-1.4.0.1.git.ec3446e-1.el7.centos.x86_64 + exit 0 Finish: rpmbuild envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm Finish: build phase for envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm INFO: Done(/tmp/tmp6bczse2a/repo/envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm) Config(child) 22 minutes 29 seconds INFO: Results and/or logs in: /var/lib/copr-rpmbuild/results INFO: Cleaning up build root ('cleanup_on_success=True') Start: clean chroot WARNING: Process ID 6527 still running in chroot. Killing... INFO: unmounting tmpfs. WARNING: tmpfs-plugin: exception while umounting tmpfs! (cwd: /home/mockbuilder) WARNING: tmpfs-plugin: exception while force umounting tmpfs! (cwd: /home/mockbuilder) ERROR: Exception(/tmp/tmp6bczse2a/repo/envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm) Config(604247-epel-7-x86_64) 22 minutes 46 seconds INFO: Results and/or logs in: /var/lib/copr-rpmbuild/results INFO: Cleaning up build root ('cleanup_on_failure=True') Start: clean chroot Traceback (most recent call last): File "/usr/libexec/mock/mock", line 480, in rebuild_generic commands.clean() File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean self.buildroot.delete() File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 609, in delete util.rmtree(self.basedir, selinux=self.selinux) File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree os.rmdir(path) OSError: [Errno 16] Device or resource busy: '/var/lib/mock/604247-epel-7-x86_64/root' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/libexec/mock/mock", line 933, in <module> main() File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 736, in main run_command(options, args, config_opts, commands, buildroot, state) File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 820, in run_command do_rebuild(config_opts, commands, buildroot, args) File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 527, in do_rebuild post=post_build, clean=clean) File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 492, in rebuild_generic commands.clean() File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean self.buildroot.delete() File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 609, in delete util.rmtree(self.basedir, selinux=self.selinux) File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree rmtree(fullname, selinux=selinux, exclude=exclude) File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace result = func(*args, **kw) File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree os.rmdir(path) OSError: [Errno 16] Device or resource busy: '/var/lib/mock/604247-epel-7-x86_64/root' Build failed ``` Version-Release number of selected component (if applicable): Mock Version: 1.4.4 How reproducible: Frequent. Perhaps 4/5 or 9/10 it fails. Steps to Reproduce: 1. build triggered from github webhook 2. 3. Actual results: some of the fedora or centos builds are failing with device-busy during cleanup Expected results: successful build, regardless of the cleanup task. Additional info:
This is because bazel shutdown does not really shut down. So it is still using buildroot and we cannot unmount it. Mock is trying to kill it: WARNING: Process ID 6527 still running in chroot. Killing... but apparently without luck. I see that we are sending just SIGTERM. I enhanced code to send SIGKILL in case SIGTERM is not enough. Commit: * a53e695 (HEAD -> devel) orphanskill: send SIGKILL when SIGTERM is not enough [RHBZ#1495214]
mock-1.4.8-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e954edb1bb
mock-1.4.8-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ac67357aa1
mock-1.4.8-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0c8682fd2a
mock-1.4.8-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e954edb1bb
mock-1.4.8-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ac67357aa1
mock-1.4.8-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0c8682fd2a
mock-1.4.8-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.4.8-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
mock-1.4.8-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.