Bug 1495214

Summary: Cleanup failed; device is busy
Product: [Fedora] Fedora Reporter: Vincent Batts <vbatts>
Component: mockAssignee: Miroslav Suchý <msuchy>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: jdisnard, mebrown, msimacek, msuchy, praiskup, williams
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mock-1.4.8-1.fc26 mock-1.4.8-1.fc27 mock-1.4.8-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-02 16:26:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vincent Batts 2017-09-25 14:06:17 UTC
Description of problem:

https://copr.fedorainfracloud.org/coprs/vbatts/envoy/builds/

Attempting to build in copr, and while the build of my RPM is successful, the cleanup of the chroot fails, resulting in a failed build.

I had opened https://bugzilla.redhat.com/show_bug.cgi?id=1493244, but was instructed to open this bug with mock as well.

end of that log shows a successful build, but failed cleanup:
```
+ cd /builddir/build/BUILD
+ cd envoy-ec3446e9b3631568db6223e9eaf37fc6ce9f6cf2
+ /usr/bin/rm -rf /builddir/build/BUILDROOT/envoy-1.4.0.1.git.ec3446e-1.el7.centos.x86_64
+ exit 0
Finish: rpmbuild envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm
Finish: build phase for envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm
INFO: Done(/tmp/tmp6bczse2a/repo/envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm) Config(child) 22 minutes 29 seconds
INFO: Results and/or logs in: /var/lib/copr-rpmbuild/results
INFO: Cleaning up build root ('cleanup_on_success=True')
Start: clean chroot
WARNING: Process ID 6527 still running in chroot. Killing...
INFO: unmounting tmpfs.
WARNING: tmpfs-plugin: exception while umounting tmpfs! (cwd: /home/mockbuilder)
WARNING: tmpfs-plugin: exception while force umounting tmpfs! (cwd: /home/mockbuilder)
ERROR: Exception(/tmp/tmp6bczse2a/repo/envoy-1.4.0.1.git.ec3446e-1.fc26.src.rpm) Config(604247-epel-7-x86_64) 22 minutes 46 seconds
INFO: Results and/or logs in: /var/lib/copr-rpmbuild/results
INFO: Cleaning up build root ('cleanup_on_failure=True')
Start: clean chroot
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 480, in rebuild_generic
    commands.clean()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 609, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/604247-epel-7-x86_64/root'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 933, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 736, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 820, in run_command
    do_rebuild(config_opts, commands, buildroot, args)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 527, in do_rebuild
    post=post_build, clean=clean)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 492, in rebuild_generic
    commands.clean()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 609, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/604247-epel-7-x86_64/root'
Build failed
```

Version-Release number of selected component (if applicable):
Mock Version: 1.4.4

How reproducible:
Frequent. Perhaps 4/5 or 9/10 it fails.

Steps to Reproduce:
1. build triggered from github webhook
2.
3.

Actual results:
some of the fedora or centos builds are failing with device-busy during cleanup

Expected results:
successful build, regardless of the cleanup task.

Additional info:

Comment 1 Miroslav Suchý 2017-12-22 10:13:05 UTC
This is because
  bazel shutdown
does not really shut down. So it is still using buildroot and we cannot unmount it. 
Mock is trying to kill it:
  WARNING: Process ID 6527 still running in chroot. Killing...
but apparently without luck.

I see that we are sending just SIGTERM. I enhanced code to send SIGKILL in case SIGTERM is not enough.

Commit:
* a53e695 (HEAD -> devel) orphanskill: send SIGKILL when SIGTERM is not enough [RHBZ#1495214]

Comment 2 Fedora Update System 2017-12-22 12:57:40 UTC
mock-1.4.8-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e954edb1bb

Comment 3 Fedora Update System 2017-12-22 12:58:21 UTC
mock-1.4.8-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ac67357aa1

Comment 4 Fedora Update System 2017-12-22 12:58:59 UTC
mock-1.4.8-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0c8682fd2a

Comment 5 Fedora Update System 2017-12-23 19:13:52 UTC
mock-1.4.8-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e954edb1bb

Comment 6 Fedora Update System 2017-12-23 19:44:00 UTC
mock-1.4.8-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ac67357aa1

Comment 7 Fedora Update System 2017-12-23 22:36:17 UTC
mock-1.4.8-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0c8682fd2a

Comment 8 Fedora Update System 2018-01-02 16:26:23 UTC
mock-1.4.8-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2018-01-02 16:53:29 UTC
mock-1.4.8-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 Fedora Update System 2018-01-09 16:18:17 UTC
mock-1.4.8-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.