1097359 – virt-sparsify hangs with 'No space left on device' while filling LV

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1097359 - virt-sparsify hangs with 'No space left on device' while filling LV

Summary: virt-sparsify hangs with 'No space left on device' while filling LV

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	libguestfs
Sub Component:
Version:	6.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Pino Toscano
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-05-13 15:53 UTC by Bryn M. Reeves
Modified:	2015-02-15 07:17 UTC (History)
CC List:	7 users (show)
Fixed In Version:	libguestfs-1.20.11-7.el6
Doc Type:	Bug Fix
Doc Text:	Cause: virt-sparsify does not check for free space available in the temporary directory. Consequence: virt-sparsify hangs if the temporary directory has not enough free space. Fix: virt-sparsify checks by default for available space before the sparsification. Result: The user is warned by default when there is not enough free space in the temporary directory.
Clone Of:
Environment:
Last Closed:	2014-10-14 06:35:23 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
output of 'virt-sparsify -v' for an el6 image with LIBVIRT_DEBUG=1 LIBVIRT_TRACE=1 (55.62 KB, text/plain) 2014-05-13 15:54 UTC, Bryn M. Reeves	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2014:1458	0	normal	SHIPPED_LIVE	libguestfs bug fix update	2014-10-14 01:10:57 UTC

Description Bryn M. Reeves 2014-05-13 15:53:12 UTC

Description of problem:
# virt-sparsify -v bmr-rhel6-vm1.img bmr-rhel6-vm1.img1
[...]
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 2f | 00 00 00 01 | 00 12 34 15 | ...
libguestfs: send_to_daemon: 84 bytes: 00 00 00 50 | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 01 | 00 00 00 00 | ...
guestfsd: main_loop: proc 47 (umount_all) took 0.11 seconds
guestfsd: main_loop: new request, len 0x50
mount -o  /dev/vg_bmrrhel6vm1/lv_root /sysroot/
[    9.939124] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 01 | 00 00 00 01 | 00 12 34 16 | ...
Fill free space in /dev/vg_bmrrhel6vm1/lv_root with zero ...
libguestfs: send_to_daemon: 52 bytes: 00 00 00 30 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 37 | 00 00 00 00 | ...
guestfsd: main_loop: proc 1 (mount) took 0.07 seconds
guestfsd: main_loop: new request, len 0x30
random filename: /sysroot//l5ij99ds.06q
◒ 60% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒════════════════════════════════════════════════════════════════⟧ 00:24
block I/O error in device 'hd0': No space left on device (28)
block I/O error in device 'hd0': No space left on device (28)
block I/O error in device 'hd0': No space left on device (28)
block I/O error in device 'hd0': No space left on device (28)
block I/O error in device 'hd0': No space left on device (28)
[ ... forever ... ]

Version-Release number of selected component (if applicable):
libguestfs-tools-c-1.20.11-2.el6.x86_64

How reproducible:
100% with the images I've tested with.

Steps to Reproduce:
1. virt-sparsify -v bmr-rhel6-vm1.img bmr-rhel6-vm1.img1

Actual results:
Command never terminates. Running verbosely produces repeating ENOSPC errors seen above.

Expected results:
Command completes or terminates with an error.

Additional info:

Comment 1 Bryn M. Reeves 2014-05-13 15:54:03 UTC

Created attachment 895191 [details]
output of 'virt-sparsify -v' for an el6 image with LIBVIRT_DEBUG=1 LIBVIRT_TRACE=1

Comment 2 Bryn M. Reeves 2014-05-13 15:58:49 UTC

Turns out this was due to the host root fs (containing /tmp) having insufficient space for the temporary image:

# du -ch /tmp/sparsifyf518c3.qcow2 
1.4G	/tmp/sparsifyf518c3.qcow2
1.4G	total

The sparsify works fine with sufficient space available in /tmp; still it'd be nice if this failed cleanly in this situation rather than simply blocking (without -v there's no way to see what's actually going on and even then it's not immediately clear where the ENOSPC is coming from).

The sparsify seems to be working fine now:

Copy to destination and make sparse ...
qemu-img convert -f qcow2 -O 'raw' '/tmp/sparsifyf518c3.qcow2' 'bmr-rhel6-vm1.img1'

Comment 4 Richard W.M. Jones 2014-05-14 08:53:40 UTC

This is essentially another instance of
https://bugzilla.redhat.com/show_bug.cgi?id=745576

Two things have improved upstream which make this less of a problem
with more recent virt-sparsify than in RHEL 6.5:

(1) virt-sparsify now checks how much space is available
before starting and can warn or give a fatal error if there
is not enough.  The relevant commits are:
7c463ac477168df5d4e4eb472ba01fa18b89c1a6
5b278937dfd5177eaafcbc41830450ef5c0a2ab8
This feature was added in 1.23.14 and as far as I remember was
not backported to any version of RHEL.

(2) virt-sparsify --in-place (in libguestfs 1.26) lets you
do in-place sparsification which doesn't require the huge
temporary file.

I'm leaving this bug open because I think we should consider
a backport of the commits in (1) to RHEL 6.

RHEL 7.1 will probably get a rebase to 1.26/1.28so we don't need
a bug for that.

Comment 5 Richard W.M. Jones 2014-06-19 08:05:39 UTC

Also ae2dd1a9e77b06ae9e7fc8ff47d8b5ea7331e0a9 is required.

Comment 7 yuliu 2014-09-04 06:10:46 UTC

Version:
Host: RHEL6.6(2.6.32-498.el6.x86_64)
Libguestfs: libguestfs-1.20.11-10.el6.x86_64


Steps:
1. fill /tmp with some files.
2. virt-sparify -v file1 file2
3. check
when the /tmp is full, some blocking info was gave out.

Result: We can verify this bug as above.

Comment 8 errata-xmlrpc 2014-10-14 06:35:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1458.html

Note You need to log in before you can comment on or make changes to this bug.