Bug 1239053

Summary:	virt-v2v error reporting when grub.conf cannot be parsed by Augeas
Product:	Red Hat Enterprise Linux 7	Reporter:	Dominic Cleal <dcleal>
Component:	libguestfs	Assignee:	Richard W.M. Jones <rjones>
Status:	CLOSED ERRATA	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	medium	Docs Contact:
Priority:	low
Version:	7.2	CC:	dgross, juzhou, mzhan, ptoscano, rjones, tzheng, virt-bugs, xiaodwan
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	V2V
Fixed In Version:	libguestfs-1.28.1-1.44.el7	Doc Type:	Bug Fix
Doc Text:	Cause: Customer had an incorrect grub configuration, but the virt-v2v error message did not explain this well. Consequence: Conversion of the guest failed, but with a less than helpful error message. Fix: The error message has been improved. Result: Conversion will still fail with an error, but the error message is actionable now.	Story Points:	---
Clone Of:	1219216	Environment:
Last Closed:	2015-11-19 07:02:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dominic Cleal 2015-07-03 11:43:44 UTC

Cloned to RHEL7 for verification that the new virt-v2v (as part of libguestfs) now reports parse errors of grub.conf from Augeas, rather than printing that the virtio kernel couldn't be found.

+++ This bug was initially created as a clone of Bug #1219216 +++

Description of problem:

   augeas tool fails to find the kernel line in /boot/grub/grub.conf if unexpected console lines are present when trying to virt-v2v a guest

Version-Release number of selected component (if applicable):

   virt-v2v-0.9.1-5.el6_5.x86_64

How reproducible:  Always

   
Steps to Reproduce:
1.  *note:  This was encountered by customer extracting a VM from an ESX environment, and adding it to a RHEV 3.5 environment, but from what I see that shouldn't matter and I was able to reproduce it by extracting a local kvm guest.

2.  Add the "serial" and "terminal" lines to the grub.conf of a VM guest to be converted.  (Examples at the bottom of this post)

3.  Attempt a virt-v2v of that VM guest:
Example:
# virt-v2v -ic qemu:///session -o rhev -os myrhev.redhat.com:/exportdomain --network rhevm kvmr6n1 


Actual results:

virt-v2v fails with the following two lines in the virt-v2v.log:

virtio configured, but no virtio kernel found at /usr/share/perl5/vendor_perl/Sys/VirtConvert/Converter/RedHat.pm line 1007.
virt-v2v: Transferring storage volume New Virtual Machine_New Virtual Machine: 161061273600 bytes


Expected results:

  Successfully create the VM for import (in this case in the RHEV 3.5 export domain.

Additional info:


Notes from support case is that one of the aug_match() calls in /usr/share/perl5/vendor_perl/Sys/VirtConvert/Converter/RedHat.pm is having problems with those two lines"


"The problem seems the be the augeas tool:

/usr/share/perl5/vendor_perl/Sys/VirtConvert/Converter/RedHat.pm:
...
 141 sub list_kernels
...
 150     # Look for a kernel, starting with the default
 151     my @paths = eval {
 152         # Get a list of all kernels
 153         my @ret = $g->aug_match("/files$grub_conf/title/kernel");

We use aug_match which does not return the value. The problem is the two lines
related to the serial console

    serial -unit=0  -speed=115200 
    terminal -timeout=10 serial console   
"


Example grub.conf with the "serial" and "terminal" lines....

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg00-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
serial -unit=0 -speed=115200
terminal -timeout=10 serial console
default=0
timeout=10
#splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-220.23.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-220.23.1.el6.x86_64 ro root=/dev/mapper/vg00-lv_root 
	initrd /initramfs-2.6.32-220.23.1.el6.x86_64.img

--- Additional comment from Dominic Cleal on 2015-05-07 10:40:45 BST ---

(In reply to Dylan Gross from comment #0)
> We use aug_match which does not return the value. The problem is the two
> lines
> related to the serial console
> 
>     serial -unit=0  -speed=115200 
>     terminal -timeout=10 serial console   
> "

Are these actually valid and working configuration directives?

The documentation suggests it must be two hyphens, not one:

https://www.gnu.org/software/grub/manual/legacy/Serial-terminal.html
https://www.gnu.org/software/grub/manual/legacy/serial.html
https://www.gnu.org/software/grub/manual/legacy/terminal.html

I see nothing there to suggest that "-unit=0" is valid.

--- Additional comment from Dominic Cleal on 2015-05-07 12:05:03 BST ---

Checked the GRUB 0.97 source code too, and it doesn't parse options with only one hyphen, which makes me believe this is a configuration error.  Augeas generally only tries to parse valid configs, not be tolerant of errors.

--- Additional comment from Dylan Gross on 2015-05-28 14:56:58 BST ---


Customer confirmed that we he did remedy those incorrect arguments with double-hyphens, the virt-v2v no longer failed.  If that is the intended behaviour of augeas when parsing, then this is probably "NOTABUG".   

Perhaps the error about not being able to match a kernel line could be more specific about failing on a different line.  Or maybe not.  I don't know enough about how the parsing is being done to know if it can get that granular.

--- Additional comment from Dominic Cleal on 2015-05-28 15:05:04 BST ---

(In reply to Dylan Gross from comment #4)
> Customer confirmed that we he did remedy those incorrect arguments with
> double-hyphens, the virt-v2v no longer failed.  If that is the intended
> behaviour of augeas when parsing, then this is probably "NOTABUG".   

Indeed.

> Perhaps the error about not being able to match a kernel line could be more
> specific about failing on a different line.  Or maybe not.  I don't know
> enough about how the parsing is being done to know if it can get that
> granular.

Yeah, it looks like virt-v2v isn't showing that Augeas hasn't managed to parse the file.  It should be possible to report Augeas' own error output (from /augeas//error), like we did here for netcf: https://lists.fedorahosted.org/pipermail/netcf-devel/2014-September/000907.html

or Puppet: https://github.com/puppetlabs/puppet/blob/4.1.0/lib/puppet/provider/augeas/augeas.rb#L327-L354

I don't know whether you'd prefer to move this bug or recreate one against v2v.

Comment 2 Richard W.M. Jones 2015-07-06 08:42:36 UTC

To try to reproduce this with new virt-v2v:

$ virt-builder centos-6

$ guestfish -a centos-6.img -i
><fs> vi /boot/grub/grub.conf

# After editing the serial and terminal lines, the file looks like below:

><fs> cat /boot/grub/grub.conf
[...comments...]
serial -unit=0 -speed=115200
terminal -timeout=10 serial console
default=0
timeout=5
title CentOS 6 (2.6.32-504.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-504.el6.x86_64 ro [...etc...]
        initrd /initramfs-2.6.32-504.el6.x86_64.img

><fs> exit

$ virt-v2v --version
virt-v2v 1.29.48fedora=23,release=1.fc23,libvirt

$ virt-v2v -i disk centos-6.img -o null
[   0.0] Opening the source -i disk centos-6.img
[   0.1] Creating an overlay to protect the source from being modified
[   0.3] Opening the overlay
[  15.9] Initializing the target -o null
[  15.9] Inspecting the overlay
[  20.3] Checking for sufficient free disk space in the guest
[  20.3] Estimating space required on target for each disk
[  20.3] Converting CentOS release 6.6 (Final) to run on KVM
virt-v2v: error: libguestfs error: aug_get: no matching node

Comment 3 Richard W.M. Jones 2015-07-06 09:52:23 UTC

Upstream fix:

https://github.com/libguestfs/libguestfs/commit/659d56db3cc4333e7410cac6720c20b3b7fa66ad

Note that after applying this fix, conversion will still fail
(which it should do - the grub configuration of the guest is
incorrect).  However the error message changes to be more explanatory:

$ virt-v2v -i disk centos-6.img -o null
[   0.0] Opening the source -i disk centos-6.img
[   0.3] Creating an overlay to protect the source from being modified
[   0.4] Opening the overlay
[  46.8] Initializing the target -o null
[  46.8] Inspecting the overlay
[  51.5] Checking for sufficient free disk space in the guest
[  51.5] Estimating space required on target for each disk
[  51.5] Converting CentOS release 6.6 (Final) to run on KVM
virt-v2v: error: no kernels were found in the grub configuration.

This probably indicates that virt-v2v was unable to parse the grub 
configuration of this guest.

Comment 5 zhoujunqin 2015-07-07 08:58:14 UTC

I can reproduce this issue with package version:
libguestfs-1.28.1-1.43.el7.x86_64
libvirt-1.2.17-1.el7.x86_64
virt-v2v-1.28.1-1.43.el7.x86_64

Steps:
1. Prepare a health guest, then add serial and terminal lines to grub.conf file:

# cat /boot/grub/grub.conf
serial -unit=0 -speed=115200
terminal -timeout=10 serial console
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux 6 (2.6.32-570.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-570.el6.x86_64 ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
	initrd /initramfs-2.6.32-570.el6.x86_64.img

2. Convert this guest to rhevm target:
# virt-v2v -o rhev -os 10.66.90.115:/vol/v2v_auto/iscsi_export --network rhevm rhel6.7-snap5 -on rhel6.7-snap5-test
[   0.0] Opening the source -i libvirt rhel6.7-snap5
[   0.0] Creating an overlay to protect the source from being modified
[   0.0] Opening the overlay
[  95.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/iscsi_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[  95.0] Inspecting the overlay
[ 107.0] Checking for sufficient free disk space in the guest
[ 107.0] Estimating space required on target for each disk
[ 107.0] Converting Red Hat Enterprise Linux Server release 6.7 Beta (Santiago) to run on KVM
virt-v2v: error: libguestfs error: aug_get: no matching node

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Then try to verify with new build:
libvirt-1.2.17-1.el7.x86_64
libguestfs-1.28.1-1.44.el7.x86_64
virt-v2v-1.28.1-1.44.el7.x86_64

Steps:

# virt-v2v -o rhev -os 10.66.90.115:/vol/v2v_auto/iscsi_export --network rhevm rhel6.7-snap5 -on rhel6.7-snap5-test
[   0.0] Opening the source -i libvirt rhel6.7-snap5
[   0.0] Creating an overlay to protect the source from being modified
[   0.0] Opening the overlay
[  92.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/iscsi_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[  92.0] Inspecting the overlay
[ 101.0] Checking for sufficient free disk space in the guest
[ 101.0] Estimating space required on target for each disk
[ 101.0] Converting Red Hat Enterprise Linux Server release 6.7 Beta (Santiago) to run on KVM
virt-v2v: error: no kernels were found in the grub configuration.

This probably indicates that virt-v2v was unable to parse the grub 
configuration of this guest.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: Error message is clear, so move this bug from ON_QA to VERIFIED.

Comment 7 errata-xmlrpc 2015-11-19 07:02:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2183.html