Bug 1495214 - Cleanup failed; device is busy
Summary: Cleanup failed; device is busy
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mock
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miroslav Suchý
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-25 14:06 UTC by Vincent Batts
Modified: 2018-01-09 16:18 UTC (History)
6 users (show)

Fixed In Version: mock-1.4.8-1.fc26 mock-1.4.8-1.fc27 mock-1.4.8-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-02 16:26:23 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1493244 0 unspecified CLOSED Cleanup failed; device is busy 2021-02-22 00:41:40 UTC

Internal Links: 1493244

Description Vincent Batts 2017-09-25 14:06:17 UTC
Description of problem:

https://copr.fedorainfracloud.org/coprs/vbatts/envoy/builds/

Attempting to build in copr, and while the build of my RPM is successful, the cleanup of the chroot fails, resulting in a failed build.

I had opened https://bugzilla.redhat.com/show_bug.cgi?id=1493244, but was instructed to open this bug with mock as well.

end of that log shows a successful build, but failed cleanup:
```
+ cd /builddir/build/BUILD
+ cd envoy-ec3446e9b3631568db6223e9eaf37fc6ce9f6cf2
+ /usr/bin/rm -rf /builddir/build/BUILDROOT/envoy-1.4.0.1.git.ec3446e-1.el7.centos.x86_64
+ exit 0
Finish: rpmbuild envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm
Finish: build phase for envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm
INFO: Done(/tmp/tmp6bczse2a/repo/envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm) Config(child) 22 minutes 29 seconds
INFO: Results and/or logs in: /var/lib/copr-rpmbuild/results
INFO: Cleaning up build root ('cleanup_on_success=True')
Start: clean chroot
WARNING: Process ID 6527 still running in chroot. Killing...
INFO: unmounting tmpfs.
WARNING: tmpfs-plugin: exception while umounting tmpfs! (cwd: /home/mockbuilder)
WARNING: tmpfs-plugin: exception while force umounting tmpfs! (cwd: /home/mockbuilder)
ERROR: Exception(/tmp/tmp6bczse2a/repo/envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm) Config(604247-epel-7-x86_64) 22 minutes 46 seconds
INFO: Results and/or logs in: /var/lib/copr-rpmbuild/results
INFO: Cleaning up build root ('cleanup_on_failure=True')
Start: clean chroot
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 480, in rebuild_generic
    commands.clean()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 609, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/604247-epel-7-x86_64/root'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 933, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 736, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 820, in run_command
    do_rebuild(config_opts, commands, buildroot, args)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 527, in do_rebuild
    post=post_build, clean=clean)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 492, in rebuild_generic
    commands.clean()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 609, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/604247-epel-7-x86_64/root'
Build failed
```

Version-Release number of selected component (if applicable):
Mock Version: 1.4.4

How reproducible:
Frequent. Perhaps 4/5 or 9/10 it fails.

Steps to Reproduce:
1. build triggered from github webhook
2.
3.

Actual results:
some of the fedora or centos builds are failing with device-busy during cleanup

Expected results:
successful build, regardless of the cleanup task.

Additional info:

Comment 1 Miroslav Suchý 2017-12-22 10:13:05 UTC
This is because
  bazel shutdown
does not really shut down. So it is still using buildroot and we cannot unmount it. 
Mock is trying to kill it:
  WARNING: Process ID 6527 still running in chroot. Killing...
but apparently without luck.

I see that we are sending just SIGTERM. I enhanced code to send SIGKILL in case SIGTERM is not enough.

Commit:
* a53e695 (HEAD -> devel) orphanskill: send SIGKILL when SIGTERM is not enough [RHBZ#1495214]

Comment 2 Fedora Update System 2017-12-22 12:57:40 UTC
mock-1.4.8-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e954edb1bb

Comment 3 Fedora Update System 2017-12-22 12:58:21 UTC
mock-1.4.8-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ac67357aa1

Comment 4 Fedora Update System 2017-12-22 12:58:59 UTC
mock-1.4.8-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0c8682fd2a

Comment 5 Fedora Update System 2017-12-23 19:13:52 UTC
mock-1.4.8-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e954edb1bb

Comment 6 Fedora Update System 2017-12-23 19:44:00 UTC
mock-1.4.8-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ac67357aa1

Comment 7 Fedora Update System 2017-12-23 22:36:17 UTC
mock-1.4.8-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0c8682fd2a

Comment 8 Fedora Update System 2018-01-02 16:26:23 UTC
mock-1.4.8-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2018-01-02 16:53:29 UTC
mock-1.4.8-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 Fedora Update System 2018-01-09 16:18:17 UTC
mock-1.4.8-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.