Bug 1294977

Summary: [abrt] mock: util.py:145:rmtree:OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-23-x86_64/root/var/cache/dnf'
Product: [Fedora] Fedora Reporter: Niki Guldbrand <niki.guldbrand>
Component: mockAssignee: Miroslav Suchý <msuchy>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: jaruga, jdisnard, kkleine, loganjerry, mebrown, Mikhail_Campos-Guadamuz, msimacek, msuchy, niki.guldbrand, praiskup, richard.kellner, robinlee.sysu, williams
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/03eedd6ea2590de5601f408ca49963283a821699
Whiteboard: abrt_hash:cefc14d18725ff3deb992091a87635738d72788d;VARIANT_ID=workstation;
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-29 11:56:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
File: backtrace
none
File: environ none

Description Niki Guldbrand 2015-12-31 12:16:37 UTC
Description of problem:
An earlier mock issue had left the var/cache/dnf directory mounted in the nspawn root

Version-Release number of selected component:
mock-1.2.14-1.fc23

Additional info:
reporter:       libreport-2.6.3
cmdline:        /usr/bin/python3 -tt /sbin/mock --dnf --update --buildsrpm --spec=zsh.spec --sources=.
executable:     /sbin/mock
kernel:         4.2.8-300.fc23.x86_64
runlevel:       N 5
type:           Python3
uid:            0

Truncated backtrace:
util.py:145:rmtree:OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-23-x86_64/root/var/cache/dnf'

Traceback (most recent call last):
  File "/sbin/mock", line 833, in <module>
    main()
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/sbin/mock", line 649, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/sbin/mock", line 732, in run_command
    do_buildsrpm(config_opts, commands, buildroot, options, args)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/sbin/mock", line 511, in do_buildsrpm
    cmd=cmd, post=None, clean=clean)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/sbin/mock", line 438, in rebuild_generic
    commands.clean()
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.4/site-packages/mockbuild/backend.py", line 72, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.4/site-packages/mockbuild/buildroot.py", line 563, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 136, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.4/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.4/site-packages/mockbuild/util.py", line 145, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-23-x86_64/root/var/cache/dnf'

Local variables in innermost frame:
exclude: ()
failed_to_handle: False
name: 'updates.solv'
mode: 33204
try_again: False
retries: 2
failed_filename: None
fullname: '/var/lib/mock/fedora-23-x86_64/root/var/cache/dnf/updates.solv'
names: []
path: '/var/lib/mock/fedora-23-x86_64/root/var/cache/dnf'
selinux: False

Comment 1 Niki Guldbrand 2015-12-31 12:16:46 UTC
Created attachment 1110764 [details]
File: backtrace

Comment 2 Niki Guldbrand 2015-12-31 12:16:48 UTC
Created attachment 1110765 [details]
File: environ

Comment 3 Robin Lee 2016-03-30 08:23:39 UTC
I met the same issue. Run mock to create a root by user1, and then try to clean the root with user2. Both user1 and user2 are mock users.

Comment 4 Mikhail Campos 2016-08-09 13:15:24 UTC
I can't reproduce the issue neither with mock-1.2.14 nor mock-1.2.18.

Do you still have the issue using mock-1.2.18?

Comment 5 Jerry James 2016-09-09 20:06:19 UTC
I just hit this issue with mock 1.2.20.  I was running mock on a remote machine, and the connection dropped during the build.  When I logged back in to the remote machine, the mock process and all build-related processes inside the chroot were gone, but /var/cache/dnf was still mounted.  As a result, I couldn't restart the build and couldn't --clean the chroot until I got a sysadmin to manually unmount that directory.

This is with mock-1.2.20-1.fc23.noarch.

Comment 6 Fedora End Of Life 2016-11-24 14:37:32 UTC
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 7 Fedora End Of Life 2016-12-20 17:30:58 UTC
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 8 Richard Kellner 2017-09-15 19:55:17 UTC
It seems I have just hit this bug as well, in Fedora 26. 

I have tried to build an rpm in mock from source rpm, and it has crashed. Unfortunately, I haven't seen the exact mock error, but I have found in journactl:

Sep 15 21:34:36 xps python3[7224]: detected unhandled Python exception in '/usr/libexec/mock/mock'

When I try to re-run the build again it ends up with an error:

INFO: mock.py version 1.4.3 starting (python version = 3.6.2)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
INFO: Start(SRPMS/python-ludolph-weather-1.0-1.fc26.src.rpm)  Config(fedora-26-x86_64)
Start: clean chroot
ERROR: Exception(SRPMS/python-ludolph-weather-1.0-1.fc26.src.rpm) Config(fedora-26-x86_64) 0 minutes 2 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-26-x86_64/result
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 933, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 736, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 820, in run_command
    do_rebuild(config_opts, commands, buildroot, args)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 527, in do_rebuild
    post=post_build, clean=clean)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 470, in rebuild_generic
    commands.clean()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 609, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-26-x86_64/root/var/cache/dnf'


Possibly other related information might be in this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1460522

Comment 9 Jun Aruga 2017-12-21 20:05:14 UTC
Hi Richard Kellner

Which Fedora package we can reproduce this issue?
I could not find python-ludolph-weather package that you showed on Fedora.

Which version of mock did you try?

The latest version is 1.4.7.

```
$ rpm -q mock
mock-1.4.7-1.fc27.noarch
```

Comment 10 Fedora End Of Life 2018-05-03 08:45:24 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 11 Jun Aruga 2018-05-07 09:38:56 UTC
> INFO: mock.py version 1.4.3 starting (python version = 3.6.2)...

Can you try the latest version of mock?

Comment 12 Fedora End Of Life 2018-05-29 11:56:16 UTC
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26
is no longer maintained, which means that it will not receive any
further security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 13 Konrad Kleine 2021-06-09 14:48:32 UTC
I'm seeing the same "device or resource error" on a mock clean every now and then on

Linux tofan 5.11.16-200.fc33.x86_64 #1 SMP Wed Apr 21 16:08:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

and mock v2.10. I'm not able to reproduce it with certainty. That's why I've switched to building with containers and rpmbuild instead.