Bug 1465556

Summary: [abrt] mock: rmtree(): util.py:160:rmtree:OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-rawhide-x86_64/root'
Product: [Fedora] Fedora Reporter: Vít Ondruch <vondruch>
Component: mockAssignee: Michael Simacek <msimacek>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 27CC: ego.cordatus, jdisnard, mebrown, msimacek, msuchy, praiskup, reto.gantenbein, williams
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/4659aed1fa8ede18423d5b94231c42d826f701f2
Whiteboard: abrt_hash:a261aae615fa1b11536dc19d95b8be47454e8e28;
Fixed In Version: mock-1.4.5-1.el7 mock-1.4.5-1.fc25 mock-1.4.5-1.fc27 mock-1.4.5-1.fc28 mock-1.4.5-1.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-13 08:43:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: cgroup
none
File: cpuinfo
none
File: environ
none
File: mountinfo
none
File: namespaces
none
File: open_fds none

Description Vít Ondruch 2017-06-27 15:35:07 UTC
Description of problem:
I am using bootstrap buildroot together with LVM. But it has still issues with mounting/unmonting the volumes correctly and it is very likely related to the rhbz#1447658

Version-Release number of selected component:
mock-1.4.2-1.fc27

Additional info:
reporter:       libreport-2.9.1
cmdline:        /usr/bin/python3 -tt /usr/libexec/mock/mock -r fedora-rawhide-x86_64 --init
crash_function: rmtree
exception_type: OSError
executable:     /usr/libexec/mock/mock
kernel:         4.11.0-1.fc27.x86_64
runlevel:       N 5
type:           Python3
uid:            0

Truncated backtrace:
util.py:160:rmtree:OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-rawhide-x86_64/root'

Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 934, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 737, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 763, in run_command
    commands.clean()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 78, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 607, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 160, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-rawhide-x86_64/root'

Local variables in innermost frame:
mode: 16877
fullname: '/var/lib/mock/fedora-rawhide-x86_64/root/opt'
name: 'opt'
names: []
failed_filename: None
failed_to_handle: False
retries: 2
try_again: False
exclude: ()
selinux: False
path: '/var/lib/mock/fedora-rawhide-x86_64/root'

Comment 1 Vít Ondruch 2017-06-27 15:35:15 UTC
Created attachment 1292393 [details]
File: backtrace

Comment 2 Vít Ondruch 2017-06-27 15:35:17 UTC
Created attachment 1292394 [details]
File: cgroup

Comment 3 Vít Ondruch 2017-06-27 15:35:18 UTC
Created attachment 1292395 [details]
File: cpuinfo

Comment 4 Vít Ondruch 2017-06-27 15:35:20 UTC
Created attachment 1292396 [details]
File: environ

Comment 5 Vít Ondruch 2017-06-27 15:35:22 UTC
Created attachment 1292397 [details]
File: mountinfo

Comment 6 Vít Ondruch 2017-06-27 15:35:23 UTC
Created attachment 1292398 [details]
File: namespaces

Comment 7 Vít Ondruch 2017-06-27 15:35:25 UTC
Created attachment 1292399 [details]
File: open_fds

Comment 8 Reto Gantenbein 2017-07-30 18:30:22 UTC
Same error here after update from mock-1.3.4-1.fc24 to mock-1.4.2-1.fc25.

site-defaults.cfg:
config_opts['plugin_conf']['root_cache_enable'] = False
config_opts['plugin_conf']['lvm_root_enable'] = True
config_opts['plugin_conf']['lvm_root_opts'] = {}
config_opts['plugin_conf']['lvm_root_opts']['volume_group'] = 'vg_system'
config_opts['plugin_conf']['lvm_root_opts']['size'] = '2G'

When running 'buildsrpm' followed by 'copyout' and then 'rebuild' the error above error would happen in the rebuild step:

INFO: mock.py version 1.4.2 starting (python version = 3.5.3)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
INFO: Start(python-pylxd.spec)  Config(fedora-25-x86_64)
Start: clean chroot
ERROR: Exception(python-pylxd.spec) Config(fedora-25-x86_64) 0 minutes 2 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-25-x86_64/result
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 934, in <module>
    main()
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 737, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 824, in run_command
    do_buildsrpm(config_opts, commands, buildroot, options, args)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 545, in do_buildsrpm
    cmd=cmd, post=None, clean=clean)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 470, in rebuild_generic
    commands.clean()
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/backend.py", line 78, in clean
    self.buildroot.delete()
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/buildroot.py", line 607, in delete
    util.rmtree(self.basedir, selinux=self.selinux)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/util.py", line 151, in rmtree
    rmtree(fullname, selinux=selinux, exclude=exclude)
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/util.py", line 160, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-25-x86_64/root'


The path /var/lib/mock/fedora-25-x86_64/root is a logical volume still mounted after the command fails.

Comment 9 Jan Kurik 2017-08-15 06:22:08 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

Comment 10 Reto Gantenbein 2017-08-16 17:12:49 UTC
Same issue happens with mock-1.4.3-1.fc25.noarch and the mentioned configuration.

However, setting the following option, seems to solve the issue for me:

config_opts['plugin_conf']['lvm_root_opts']['umount_root'] = True

Comment 11 Vít Ondruch 2017-08-17 05:28:03 UTC
(In reply to Reto Gantenbein from comment #10)
Well, the true is that the umount_root does not work correctly (bug #1447658), so I could change it to True, but I don't want to ....

Comment 12 Michael Simacek 2017-08-17 07:44:42 UTC
I'll submit a PR

Comment 13 Michael Simacek 2017-10-13 08:43:42 UTC
This should be fixed in current mock already by https://github.com/rpm-software-management/mock/pull/114. Please reopen if it's not the case.