Bug 1289327 - failure when deleting disk image: cannot unlink file 'XXX': Permission denied
Summary: failure when deleting disk image: cannot unlink file 'XXX': Permission denied
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 23
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1316173 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-07 21:16 UTC by Martín Cigorraga
Modified: 2016-03-24 08:39 UTC (History)
18 users (show)

Fixed In Version: libvirt-1.2.18.2-3.fc23
Clone Of:
Environment:
Last Closed: 2016-03-22 19:53:23 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1306536 0 unspecified CLOSED SLES12-Libvirt: virsh vol-delete fails when deleting disk image: cannot unlink file 'XXX': Permission denied 2021-02-22 00:41:40 UTC

Internal Links: 1306536

Description Martín Cigorraga 2015-12-07 21:16:18 UTC
Description of problem:
Virt-manager crashes without completing the operation when trying to remove a VM qcow2 disk image.

Version-Release number of selected component (if applicable):
1.3.0

How reproducible:
Always.

Steps to Reproduce:
1. Create a new VM (QEMU/KVM here);
2. Remove the VM while trying to also remove the disk image.
3.

Actual results:
Virt-manager will crash without removing the .qcow2 image, which has to be erased manually.

Expected results:
The VM disk image is removed.

Additional info:
The provided error message follows:
(Dialog window)
Error refreshing volume 'openSUSE-Tumbleweed.qcow2': cannot unlink file '/var/lib/libvirt/images/openSUSE-Tumbleweed.qcow2': Success

(Clicking on Details, same dialog)
"Error refreshing volume 'openSUSE-Tumbleweed.qcow2': cannot unlink file '/var/lib/libvirt/images/openSUSE-Tumbleweed.qcow2': Success

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 90, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 126, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/storagelist.py", line 731, in cb
    vol.delete()
  File "/usr/share/virt-manager/virtManager/storagepool.py", line 77, in delete
    self._backend.delete(0)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3348, in delete
    if ret == -1: raise libvirtError ('virStorageVolDelete() failed', vol=self)
libvirtError: cannot unlink file '/var/lib/libvirt/images/openSUSE-Tumbleweed.qcow2': Success"

Comment 1 Cole Robinson 2015-12-08 17:48:57 UTC
The error message mentioning 'refresh' is incorrect, I fixed it upstream now:

commit f10bbfe5b3b6e5a9c773a2ce0a8824d252ba3786
Author: Cole Robinson <crobinso>
Date:   Tue Dec 8 12:44:59 2015 -0500

    storagelist: Fix error message for vol delete


But that error is coming from libvirt and I can't tell what the issue is... it should be reporting a valid errno.

What version of libvirt are you using? rpm -q libvirt-daemon
Is this reliably reproducible?

Comment 2 Martín Cigorraga 2015-12-08 17:52:13 UTC
Hi Cole,

libvirt-daemon: libvirt-daemon-1.2.18.1-2.fc23.x86_64libvirt-daemon-1.2.18.1-2.fc23.x86_64

Yes, this error is a standard, happens every time I want to remove a disk image.

Comment 3 Cole Robinson 2015-12-08 17:57:04 UTC
Thanks.

Did this work in the past?
What kind of filesystem is /var/lib/libvirt/images on? Or provide 'findmnt' output if you aren't sure
What kernel are you running? uname -r

Comment 4 Martín Cigorraga 2015-12-08 18:17:21 UTC
Np, thanks to you!

Well, I have been experiencing this error since F22 IIRC.
The FS is Btrfs and kernel is latest 4.2.6-301.

Comment 5 Cole Robinson 2015-12-08 18:29:41 UTC
The libvirt code here is very basic, just calling unlink(filename) and trying to report the strerror in an error message. The fact that you are using btrfs makes me thing that this is an fs issue and it's returning failure for the unlink(2) call but not setting an errno

Does the file actually end up deleted, and virt-manager just throws an error? Or does the file remain on disk?
If the file remains, does this work? (warning: it will delete the file if it succeeds): python -c 'import os; print os.unlink("/path/to/your/file")'
Also interested in the output when trying to unlink a file that doesn't exist: python -c 'import os; print os.unlink("/var/tmp/I-DONT-EXIST")'

Comment 6 Martín Cigorraga 2015-12-08 18:39:56 UTC
Q: "Does the file actually end up deleted, and virt-manager just throws an error? Or does the file remain on disk?"
A: Virt-manager throws the error and the file *remains* on disk, I have to manually delete it.

Q: # python -c 'import os; print os.unlink("/var/lib/libvirt/images/openSUSE-Leap_42.qcow2")'
A: Gives back a 'None' message and this time the file is effectively removed.

Q: # python -c 'import os; print os.unlink("/var/tmp/I-DONT-EXIST")'
A:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: [Errno 2] No such file or directory: '/var/tmp/I-DONT-EXIST'

(The Python interpreter version is 2.7.10)

Comment 7 Cole Robinson 2015-12-08 20:02:16 UTC
hmm okay. let's see if we can simplify reproducing it.

Let's see if just creating a storage volume with libvirt, then deleting it immediately triggers:

$ sudo virsh vol-create-as --pool default test.qcow2 --format qcow2 --capacity 20G --prealloc-metadata
$ sudo virsh vol-delete --pool default test.qcow2


If that doesn't work, can you try creating a VM with virt-manager then deleting it immediately? Run virt-manager --debug, use whatever VM settings you used to create the original VM you referenced in comment #0, but once the 'New VM' wizard closes, immediately delete the VM + storage, and see if it reproduces. If so, attach the --debug output here

Comment 8 Martín Cigorraga 2015-12-08 20:24:11 UTC
Q: $ sudo virsh vol-create-as --pool default test.qcow2 --format qcow2 --capacity 20G --prealloc-metadata
A: Vol test.qcow2 created

Q: $ sudo virsh vol-delete --pool default test.qcow2
A: Vol test.qcow2 deleted

It does work. Then I tried this:
1. Launch virt-manager (from GUI, it asks for my credentials as usual);
2. Open an stopped VM and choose to add a new storage --> Went to the storage pool;
3. Re-create the test.qcow2 volumen from the command line --> Refresh the storage pool listing in virt-manager => the new test.qcow2 storage appeared as expected;
4. Remove the newly created storage test.qcow2 from virt-manager interface: it deleted the storage as expected!

Do you want me to try running virt-manager with the debug flag enabled?

Comment 9 Cole Robinson 2015-12-08 21:12:14 UTC
No the --debug flag is only going to be interesting if you can reproduce the problem.

So since those virt-manager steps didn't reproduce the issue, can you distill the bug down to a reproducible set of steps? Maybe the VM disk image needs to be written to before it triggers the issue.

Comment 10 Martín Cigorraga 2015-12-08 22:34:59 UTC
I just reproduced the issue this way:

1. Create a new VM from an ISO file (in this case the Fedora 23 Security spin);
2. Once the VM is created it will automatically boot -> once the guest OS is up and running, shut the system down via Xfce's menu and then stop the VM from virt-manager's menu;
3. Right-click over the newly created VM (it shows up here as fedora22) -> Delete

I got:
"Errors encountered while removing certain storage devices.

cannot unlink file '/var/lib/libvirt/images/fedora22.qcow2': Success
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/delete.py", line 182, in _async_delete
    self._async_delete_path(conn, path, meter)
  File "/usr/share/virt-manager/virtManager/delete.py", line 228, in _async_delete_path
    vol.delete(0)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3348, in delete
    if ret == -1: raise libvirtError ('virStorageVolDelete() failed', vol=self)
libvirtError: cannot unlink file '/var/lib/libvirt/images/fedora22.qcow2': Success"

[root@laptop images]# cd /var/lib/libvirt/images/
[root@laptop images]# ll
total 11114668
-rw-r--r--. 1 root root   9665380352 Dec  1 20:40 CentOS_7-Atomic_Host.qcow2
-rw-r--r--. 1 root root  21478375424 Dec  8 19:24 fedora22.qcow2
-rw-r--r--. 1 root root 107384668160 Dec  1 16:41 Whonix-Gateway.qcow2
-rw-r--r--. 1 root root 107384668160 Dec  1 16:41 Whonix-Workstation.qcow2
[root@laptop images]#

Comment 11 André Martins 2016-01-17 05:43:01 UTC
@Cole Robinson, this seems to be related:
https://bugzilla.redhat.com/show_bug.cgi?id=1289327

Comment 12 André Martins 2016-01-17 05:44:29 UTC
(In reply to André Martins from comment #11)
> @Cole Robinson, this seems to be related:
> https://bugzilla.redhat.com/show_bug.cgi?id=1289327

Sorry, bad copy-paste. This link -> https://bugzilla.redhat.com/show_bug.cgi?id=1293804

Comment 13 Cole Robinson 2016-02-11 19:28:42 UTC
So the 'Success' error is fixed upstream:

commit cb19cff468432e55366014658f405066ce06c2f2
Author: John Ferlan <jferlan>
Date:   Wed Sep 30 17:37:27 2015 -0400

    virfile: Fix error path for forked virFileRemove

But I think the root issue is still unresolved.

Comment 14 Cole Robinson 2016-03-09 15:32:50 UTC
The root issue was discovered and there's a patch proposed upstream, but it's not acceptable as-is:

https://www.redhat.com/archives/libvir-list/2016-February/msg01312.html

Comment 15 Cole Robinson 2016-03-09 15:33:35 UTC
*** Bug 1316173 has been marked as a duplicate of this bug. ***

Comment 16 Cole Robinson 2016-03-09 17:40:13 UTC
Patches posted:

https://www.redhat.com/archives/libvir-list/2016-March/msg00310.html

Comment 17 Fedora Update System 2016-03-17 22:30:12 UTC
libvirt-1.2.18.2-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-399269e155

Comment 18 Fedora Update System 2016-03-19 01:23:59 UTC
libvirt-1.2.18.2-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-399269e155

Comment 19 Fedora Update System 2016-03-22 19:52:54 UTC
libvirt-1.2.18.2-3.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 20 Vít Ondruch 2016-03-24 08:39:25 UTC
I can confirm that the latest libvirt fixes the issues for me:

$ rpm -q libvirt-daemon
libvirt-daemon-1.3.2-3.fc25.x86_64


Note You need to log in before you can comment on or make changes to this bug.