Bug 1239154

Summary: appliance fails to start with "supermin: ext2fs_file_write: /var/log/tallylog: Could not allocate block in ext2 filesystem"
Product: Red Hat Enterprise Linux 7 Reporter: Prasad Mukhedkar <pmukhedk>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.3CC: leiwang, linl, pedro.sigwald, pmukhedk, ptoscano, rjones, wshi, xchen
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libguestfs-1.31.7-2.el7.x86_64 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 17:54:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1218766    
Bug Blocks: 1288337, 1301891    

Description Prasad Mukhedkar 2015-07-03 19:21:00 UTC
Description of problem:

'virt-df <instance>' command fails with "supermin: ext2fs_file_write: /var/log/tallylog: Could not allocate block in ext2 filesystem"
error. 

# virt-df instance-000155e2
libguestfs: error: /usr/bin/supermin5 exited with error status 1.
To see full error messages you may need to enable debugging.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs

Errors reported in debug logs :

------
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.rxy85058/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
**supermin: ext2fs_file_write: /var/log/tallylog: Could not allocate block in ext2 filesystem**
**supermin: failure: ext2fs_file_write**
**libguestfs: error: /usr/bin/supermin5 exited with error status 1, see debug messages above***
libguestfs: closing guestfs handle 0x7f8b23aaf400 (state 0)
----


Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 7.1

Name        : libguestfs-tools
Arch        : noarch
Epoch       : 1
Version     : 1.28.1
Release     : 1.18.el7
Size        : 30 k
Repo        : installed

Name        : supermin
Arch        : x86_64
Version     : 5.1.10
Release     : 1.2.el7
Size        : 217 k
How reproducible:
always on particular system at a customer end.

Additional info:
The issue is reporducible on some particular systems only. It is not on 
all the identical system.

Comment 2 Richard W.M. Jones 2015-07-03 20:30:21 UTC
If you look into /var/log on the customer system, you should find
that one file is particularly large.  It may be /var/log/tallylog,
or it might be another file from /var/log.

Having a listing of

  ls -l /var/log

would be useful.

Once you've identified the large file, you can work around the problem
by editing /usr/lib64/guestfs/supermin.d/excludefiles and add the
following line:

-/var/log/tallylog

(Or if another file is the large file, put that filename instead).

Note the filename is preceeded by a '-' character, and there must
be no extra spaces, blank lines, etc in the file.

Comment 5 Prasad Mukhedkar 2015-07-09 06:44:17 UTC
Created attachment 1050107 [details]
ls -lR /var/log output

Comment 7 Richard W.M. Jones 2015-07-09 09:01:11 UTC
Patch posted upstream:
https://www.redhat.com/archives/libguestfs/2015-July/msg00092.html

Prasad, could you try (or get the customer to try) this:

Edit /usr/lib64/guestfs/supermin.d/excludefiles

Remove all lines from the file that begin with
-/var/log/

Add a single line at the end of the file:
-/var/log/*

(note it must have a leading minus sign, and no extra spaces
or blank lines).

Run 'libguestfs-test-tool' and see if the failure is fixed.

Comment 11 Richard W.M. Jones 2015-08-10 13:35:33 UTC
Upstream commit is 95bb1916eb1b2f6f9e9575cf4bf92501292bf545.

Unfortunately we didn't manage to collect the debug information
required to ensure this fix is the correct one.  Therefore I am
moving this to RHEL 7.3.

This will be automatically picked up by the planned rebase
for RHEL 7.3 (bug 1218766).

Comment 12 Richard W.M. Jones 2015-09-18 07:43:04 UTC
*** Bug 1264241 has been marked as a duplicate of this bug. ***

Comment 17 Xianghua Chen 2016-07-26 07:23:01 UTC
Hi Rich,

I can't reproduce this bug with low version libguestfs:
libguestfs-1.28.1-1.55.el7_2.4.x86_64
libguestfs-tools-1.28.1-1.55.el7_2.4.noarch
supermin5-5.1.10-1.2.el7.x86_64

So I can't confirm if it has been modified now. From the above comments, I could get limited info about customer env, this my steps :
1. Prepare a RHEL7 host.
2. Create a big log file:
   # dd if=/dev/zero of=/var/log/bigfile count=10240 bs=1M
2. Install a RHEL7 guest image with ext2, and start it.
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    rhel7.1-ext2                   running
3. 
# virt-df -d rhel7.1-ext2
Filesystem                           1K-blocks       Used  Available  Use%
rhel7.1-ext2:/dev/sda1                  495844     104564     365680   22%
rhel7.1-ext2:/dev/rhel/root           18578172    1706384   15928072   10%


Can you help to tell me what else should I do to prepare for the env? Or just tell me how to verify this bug in other way? Thank you very much:)

Comment 18 Richard W.M. Jones 2016-07-27 11:13:06 UTC
(In reply to Xianghua Chen from comment #17)
> Hi Rich,
> 
> I can't reproduce this bug with low version libguestfs:
> libguestfs-1.28.1-1.55.el7_2.4.x86_64
> libguestfs-tools-1.28.1-1.55.el7_2.4.noarch
> supermin5-5.1.10-1.2.el7.x86_64
> 
> So I can't confirm if it has been modified now. From the above comments, I
> could get limited info about customer env, this my steps :
> 1. Prepare a RHEL7 host.
> 2. Create a big log file:
>    # dd if=/dev/zero of=/var/log/bigfile count=10240 bs=1M

I think if you create a massive /var/log/tallylog (instead
of bigfile) on the host then you should be able to reproduce this.

And then just run libguestfs-test-tool.

There is no need to create any guests.

Comment 19 Xianghua Chen 2016-07-28 10:59:40 UTC
Verified with packages:
libguestfs-1.32.6-1.el7.x86_64
supermin5-5.1.16-4.el7.x86_64

Steps:
1. Prepare a RHEL7 host.
2. Create a big log file:
   # dd if=/dev/zero of=/var/log/tallylog count=10240 bs=1M
3. # libguestfs-test-tool

Command finished successfully, there is no error occurred like:
supermin: ext2: copying files from host filesystem
supermin: error: statvfs: No space left on device: /var/tmp/.guestfs-0/appliance.d.kjkoazrb/root
libguestfs: error: /usr/bin/supermin5 exited with error status 1, see debug messages above

Comment 21 errata-xmlrpc 2016-11-03 17:54:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2576.html