Bug 1776956

Summary: [abrt] mock: logOutput(): util.py:265:rmtree:OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-rawhide-i386/root/proc/filesystems'
Product: [Fedora] Fedora Reporter: Mikhail <mikhail.v.gavrilov>
Component: mockAssignee: Copr Team <copr-team>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: jdisnard, jkeating, loganjerry, mebrown, msuchy, philip.wyett, praiskup, williams
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/1d3273bce1ff59ecb065804daa9d7c7d734af847
Whiteboard: abrt_hash:6ca0ddb74edde114335136abbfcb482adcae7ede;VARIANT_ID=workstation;
Fixed In Version: mock-2.2-1.fc32 mock-2.2-1.fc31 mock-2.2-1.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-07 05:05:04 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: cpuinfo
none
File: environ
none
File: mountinfo
none
File: namespaces
none
File: open_fds none

Description Mikhail 2019-11-26 16:16:37 UTC
Version-Release number of selected component:
mock-1.4.21-1.fc32

Additional info:
reporter:       libreport-2.11.3
cgroup:         0::/user.slice/user-1000.slice/user/gnome-terminal-server.service
cmdline:        /usr/bin/python3 -tt /usr/libexec/mock/mock -r fedora-rawhide-i386 --resultdir /home/mikhail/packaging-work/mesa/results_mesa/19.3.0~rc4/1.fc32 --rebuild /home/mikhail/packaging-work/mesa/mesa-19.3.0~rc4-1.fc32.src.rpm
crash_function: logOutput
exception_type: KeyboardInterrupt
executable:     /usr/libexec/mock/mock
interpreter:    python3-3.8.0-1.fc32.x86_64
kernel:         5.4.0-0.rc8.git0.2.fc32.x86_64
runlevel:       N 5
type:           Python3
uid:            0

Truncated backtrace:
#1 [/usr/lib/python3.8/site-packages/mockbuild/util.py:551] logOutput
#2 [/usr/lib/python3.8/site-packages/mockbuild/util.py:705] do_with_status
#3 [/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py:95] trace
#4 [/usr/lib/python3.8/site-packages/mockbuild/util.py:652] do
#5 [/usr/lib/python3.8/site-packages/mockbuild/plugins/selinux.py:106] _selinuxDoYum
#6 [/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py:95] trace
#7 [/usr/lib/python3.8/site-packages/mockbuild/package_manager.py:154] execute
#8 [/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py:95] trace
#9 [/usr/lib/python3.8/site-packages/mockbuild/package_manager.py:195] builddep
#10 [/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py:95] trace

Comment 1 Mikhail 2019-11-26 16:16:41 UTC
Created attachment 1639906 [details]
File: backtrace

Comment 2 Mikhail 2019-11-26 16:16:43 UTC
Created attachment 1639907 [details]
File: cpuinfo

Comment 3 Mikhail 2019-11-26 16:16:44 UTC
Created attachment 1639908 [details]
File: environ

Comment 4 Mikhail 2019-11-26 16:16:46 UTC
Created attachment 1639909 [details]
File: mountinfo

Comment 5 Mikhail 2019-11-26 16:16:47 UTC
Created attachment 1639910 [details]
File: namespaces

Comment 6 Mikhail 2019-11-26 16:16:49 UTC
Created attachment 1639911 [details]
File: open_fds

Comment 7 Jerry James 2020-01-18 23:04:56 UTC
To reproduce this, simply start a mock build, wait a couple of seconds, then hit Control-C.  Now try to start the mock build again and you get this error.  The user cannot fix it.  Mock doesn't clean up the problem itself, nor does mock --clean, and the user doesn't have permission to unmount the problematic filesystem.  Only root can unmount the filesystem so that the buildroot is usable again.

Note that recently, I've found myself needing to sudo umount *two* filesystems:

<mock-root>/root/proc/filesystems
<mock-root>/root/var/cache/dnf

It would be great if mock --clean, at least, could do that.

Comment 8 Ben Cotton 2020-02-11 17:33:22 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 9 Pavel Raiskup 2020-04-02 11:58:18 UTC
I believe those two, especially the second ...

https://github.com/rpm-software-management/mock/commit/2ecc9071df3cf2ceb10765c65abc8a8d82414b57
https://github.com/rpm-software-management/mock/commit/2185a51110e7129c90b61ac3abd7cefcd6c458da

... fixed these kind of issues.

My explanation would that

-> we killed `dnf install` command in the middle of the transaction
-> some background process was kept working hard on background
-> the process had opened some file descriptors that refused mock to
   umount all the mountpoins
-> mock failed hard, and
-> the mount layout ended up in inconsistent state enough so even
   further `mock --scrub=all` failed.

I'm attaching this into 2.2 bodhi update.  Please reopen if you see the
same problems with mock 2.2+.

Comment 10 Fedora Update System 2020-04-02 11:59:24 UTC
FEDORA-2020-fba9845e22 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-fba9845e22

Comment 11 Fedora Update System 2020-04-02 11:59:49 UTC
FEDORA-2020-6b7c342fb4 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-6b7c342fb4

Comment 12 Fedora Update System 2020-04-02 12:01:02 UTC
FEDORA-2020-85df0014c1 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2020-85df0014c1

Comment 13 Fedora Update System 2020-04-03 19:56:06 UTC
FEDORA-2020-fba9845e22 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-fba9845e22`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-fba9845e22

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2020-04-03 20:35:06 UTC
FEDORA-2020-85df0014c1 has been pushed to the Fedora 30 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-85df0014c1`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-85df0014c1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 15 Fedora Update System 2020-04-03 20:49:35 UTC
FEDORA-2020-6b7c342fb4 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-6b7c342fb4`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-6b7c342fb4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2020-04-07 05:05:04 UTC
FEDORA-2020-fba9845e22 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 17 Fedora Update System 2020-04-07 13:16:01 UTC
FEDORA-2020-fba9845e22 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 18 Fedora Update System 2020-04-20 22:14:47 UTC
FEDORA-2020-6b7c342fb4 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 19 Fedora Update System 2020-04-20 22:31:59 UTC
FEDORA-2020-85df0014c1 has been pushed to the Fedora 30 stable repository.
If problem still persists, please make note of it in this bug report.