Bug 1418283 - virt-v2v: appliance runs out of memory running setfiles command
Summary: virt-v2v: appliance runs out of memory running setfiles command
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On: 1359086
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-01 12:56 UTC by Pawan kumar Vilayatkar
Modified: 2017-08-01 22:11 UTC (History)
9 users (show)

Fixed In Version: libguestfs-1.36.1-1.el7
Doc Type: Bug Fix
Doc Text:
Cause: when relabelling SELinux guests, the setfiles utility could require more memory (RAM) than needed Consequence: setfiles using all the memory of the libguestfs appliance (helper VM), and being killed because of that, causing the conversion to fail Fix: virt-v2v reserves more memory when launching its appliance Result: the relabelling operation can complete successfully, and so the conversion of SELinux guests
Clone Of:
Environment:
Last Closed: 2017-08-01 22:11:26 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2023 normal SHIPPED_LIVE libguestfs bug fix and enhancement update 2017-08-01 19:32:01 UTC

Description Pawan kumar Vilayatkar 2017-02-01 12:56:54 UTC
Description of problem:

While converting a VM using virt-v2v fails with below :
"[ 1147.056078] Out of memory: Kill process 8209 (setfiles) score 876 or sacrifice child
[ 1147.056994] Killed process 8209 (setfiles) total-vm:795640kB, anon-rss:711884kB, file-rss:0kB,
guestfsd: error: Can't stat exclude path "/sysroot/selinux", No such file or directory - ignoring.: Success
libguestfs: trace: v2v: selinux_relabel = -1 (error)
virt-v2v: error: libguestfs error: selinux_relabel: Can't stat exclude path 
"/sysroot/selinux", No such file or directory - ignoring.: Success


Version-Release number of selected component (if applicable):

virt-v2v-1.32.7-3.el7_3.2.x86_64                            Thu Jan 26 09:56:01 2017
libvirt-2.0.0-10.el7_3.4.x86_64                             Thu Jan 26 10:21:47 2017
libvirt-client-2.0.0-10.el7_3.4.x86_64                      Thu Jan 26 09:55:50 2017
libguestfs-1.32.7-3.el7_3.2.x86_64                          Thu Jan 26 09:56:00 2017
libguestfs-tools-c-1.32.7-3.el7_3.2.x86_64                  Thu Jan 26 09:56:01 2017




Actual results:

"LIBGUESTFS_DEBUG=1 virt-v2v -v -x -o rhev -n netecodmz -os kvm191m103:/opt/rhev/export2 satellite.ecofis.de 2>&1 | tee /root/v2v_satellite_002.log"

[ 1147.048860] 183 total pagecache pages
[ 1147.049302] 0 pages in swap cache
[ 1147.049693] Swap cache stats: add 0, delete 0, find 0/0
[ 1147.050322] Free swap  = 0kB
[ 1147.050667] Total swap = 0kB
[ 1147.051023] 204698 pages RAM
[ 1147.051373] 0 pages HighMem/MovableOnly
[ 1147.051821] 7398 pages reserved
[ 1147.052212] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[ 1147.053132] [  100]     0   100    11399      143      22        0         -1000 systemd-udevd
[ 1147.054146] [  182]     0   182    31263      176      32        0             0 guestfsd
[ 1147.055110] [ 8209]     0  8209   198910   177971     362        0             0 setfiles
[ 1147.056078] Out of memory: Kill process 8209 (setfiles) score 876 or sacrifice child
[ 1147.056994] Killed process 8209 (setfiles) total-vm:795640kB, anon-rss:711884kB, file-rss:0kB, shmem-rss:0kB
guestfsd: error: Can't stat exclude path "/sysroot/selinux", No such file or directory - ignoring.: Success
libguestfs: trace: v2v: selinux_relabel = -1 (error)
virt-v2v: error: libguestfs error: selinux_relabel: Can't stat exclude path 
"/sysroot/selinux", No such file or directory - ignoring.: Success
umount '/tmp/v2v.xIBg9W'
rm -rf '/var/tmp/null.u7Q4NG'
libguestfs: trace: v2v: close




Additional info:

Attaching virt-v2v logs.

Comment 1 Pawan kumar Vilayatkar 2017-02-01 12:59:18 UTC
Created attachment 1246644 [details]
virt-v2v.log

Comment 3 Richard W.M. Jones 2017-02-01 13:33:48 UTC
Before running virt-v2v, can you set this environment variable and
see if it fixes the problem:

export LIBGUESTFS_MEMSIZE=2048

Comment 4 Pawan kumar Vilayatkar 2017-02-02 09:33:44 UTC
Hello Richard,

Setting the environment variable resolved the problem. 
I am closing the bugzilla

Thanks

Comment 6 Richard W.M. Jones 2017-02-02 09:37:59 UTC
Thanks for testing this.  We have an upstream fix which
increases the memory available to the appliance:

https://github.com/libguestfs/libguestfs/commit/2b5173712ba07e3e89926c82a236390a4ea6626a

Please leave this bug open so it can be included in RHEL 7.4.

Comment 8 Richard W.M. Jones 2017-02-21 20:05:17 UTC
Reproducer:

$ virt-builder fedora-25 --size 10G --output test.img
$ guestfish -a test.img -i fill-dir /etc 4000000
$ virt-v2v -i disk test.img -o null

Both the guestfish and virt-v2v commands may take a very long
time to run.  If you can't reproduce the bug, try increasing
the constant (4000000) to something even larger.

Notice that you may see various different errors from
selinux_relabel, because it can run out of memory in slightly
different places.  However as long as it's selinux_relabel
which is breaking, then you are seeing the error.

This bug (which is a bug in SELinux) appears to be fixed
in Fedora 25+

Comment 11 kuwei@redhat.com 2017-03-14 05:03:12 UTC
Verify with below bulids:
virt-v2v-1.36.2-1.el7.x86_64
libvirt-3.1.0-2.el7.x86_64
qemu-kvm-rhev-2.8.0-6.el7.x86_64

steps:(same as comment 8):
$ virt-builder fedora-25 --size 10G --output test.img
$ guestfish -a test.img -i fill-dir /etc 4000000
# virt-v2v -i disk test.img -o null
[   0.0] Opening the source -i disk test.img
[   0.1] Creating an overlay to protect the source from being modified
[   0.3] Initializing the target -o null
[   0.3] Opening the overlay
[   1.5] Inspecting the overlay
[   6.1] Checking for sufficient free disk space in the guest
[   6.1] Estimating space required on target for each disk
[   6.1] Converting Fedora 25 (Twenty Five) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown 
device "vda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[ 708.8] Mapping filesystem data to avoid copying unused and blank areas
[ 708.9] Closing the overlay
[ 709.0] Checking if the guest needs BIOS or UEFI to boot
[ 709.0] Assigning disks to buses
[ 709.0] Copying disk 1/1 to /var/tmp/null.K51C7B/sda (raw)
    (100.00/100%)
[ 725.2] Creating output metadata
[ 725.2] Finishing off

Hi,rjones,from your comment 8 reproduce steps ,the  conversion is successfully . But i am not sure if it can work better in customer Environment.

Comment 12 Richard W.M. Jones 2017-03-14 08:50:28 UTC
The real bug is a memory leak in the setfiles program.  By increasing
the memory available to the program, we can make it fail after
a longer time (hopefully much longer), but there will always be a
case where a disk is so complex that it triggers the memory leak.

Apparently we cannot fix the actual setfiles program because that's
too complex or something.

Anyway, I think we should move this to VERIFIED.

Comment 13 kuwei@redhat.com 2017-03-14 08:59:12 UTC
From comment 11 and comment 12,move the bug from ON_QA to VERIFIED

Comment 14 kuwei@redhat.com 2017-03-14 11:10:46 UTC
(In reply to kuwei@redhat.com from comment #11)
> Verify with below bulids:
> virt-v2v-1.36.2-1.el7.x86_64
> libvirt-3.1.0-2.el7.x86_64
> qemu-kvm-rhev-2.8.0-6.el7.x86_64
> 
> steps:(same as comment 8):
> $ virt-builder fedora-25 --size 10G --output test.img
> $ guestfish -a test.img -i fill-dir /etc 4000000
> # virt-v2v -i disk test.img -o null


Try to builder a rhel7.3 geust to convert as the steps:
steps::
$ virt-builder rhel-7.3 --size 10G --output test.img
$ guestfish -a test.img -i fill-dir /etc 4000000
# virt-v2v -i disk test.img -o null
[   0.0] Opening the source -i disk test.img
[   0.1] Creating an overlay to protect the source from being modified
[   0.3] Initializing the target -o null
[   0.3] Opening the overlay
[   1.5] Inspecting the overlay
[  10.2] Checking for sufficient free disk space in the guest
[  10.2] Estimating space required on target for each disk
[  10.2] Converting Red Hat Enterprise Linux Server 7.3 (Maipo) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown 
device "vda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[ 722.9] Mapping filesystem data to avoid copying unused and blank areas
[ 723.7] Closing the overlay
[ 723.7] Checking if the guest needs BIOS or UEFI to boot
[ 723.7] Assigning disks to buses
[ 723.7] Copying disk 1/1 to /var/tmp/null.nUCcAd/sda (raw)
    (100.00/100%)
[ 741.7] Creating output metadata
[ 741.7] Finishing off

Result: The conversion successfully too.

Comment 15 errata-xmlrpc 2017-08-01 22:11:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2023


Note You need to log in before you can comment on or make changes to this bug.