Bug 1401474

Summary:	Importing VMs from VMware is failing with error "Inspection field 'i_arch' was 'unknown'"
Product:	Red Hat Enterprise Linux 7	Reporter:	nijin ashok <nashok>
Component:	libguestfs	Assignee:	Richard W.M. Jones <rjones>
Status:	CLOSED ERRATA	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	high	Docs Contact:	Jiri Herrmann <jherrman>
Priority:	urgent
Version:	7.3	CC:	jsuchane, juzhou, kshukla, kuwei, mtessun, mxie, mzhan, nashok, pasik, ptoscano, snagar, tzheng, xchen, xiaodwan
Target Milestone:	rc	Keywords:	ZStream
Target Release:	7.4
Hardware:	All
OS:	All
Whiteboard:	v2v
Fixed In Version:	libguestfs-1.36.1-1.el7	Doc Type:	Bug Fix
Doc Text:	The libguestfs tools now correctly handle guests where `/usr/` is not on the same partition as root Previously, the libguestfs library did not recognize the guest operating system when the `/usr/` directory was not located on the same partition as the root directory. As a consequence, multiple libguestfs tools, such as the virt-v2v utility, did not perform as expected when used on such guests. This update ensures that libguestfs recognizes guest operating systems when `/usr/` is not on the same partition as root. As a result, the affected libguestfs tools perform as expected.	Story Points:	---
Clone Of:
Clones:	1406289 (view as bug list)		Environment:
Last Closed:	2017-08-01 22:11:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1359086
Bug Blocks:	1395265, 1401400, 1406289

Description nijin ashok 2016-12-05 11:45:45 UTC

Description of problem:

Importing VMs from VMware is failing with error "'i_arch' was 'unknown'". Issue is with RHEL 7 VMs with /usr in a different partition.

We run "file -zb" command on binaries to determine the architecture of the VMs. In recent versions of RHEL 7, the /bin is a symbolic link to /usr/bin. Since /usr is in different partition, finding the binaries are failing.

==
commandrvf: mount -o ro /dev/rhel/root /sysroot/^M

libguestfs: trace: v2v: is_file "/bin/bash" "followsymlinks:true"
guestfsd: main_loop: proc 67 (download) took 0.00 seconds^M
guestfsd: main_loop: new request, len 0x3c^M
libguestfs: trace: v2v: is_file = 0

libguestfs: trace: v2v: is_file "/bin/ls" "followsymlinks:true"
guestfsd: main_loop: proc 37 (is_file) took 0.00 seconds^M
guestfsd: main_loop: new request, len 0x38^M
libguestfs: trace: v2v: is_file = 0

libguestfs: trace: v2v: is_file "/bin/echo" "followsymlinks:true"
guestfsd: main_loop: proc 37 (is_file) took 0.00 seconds^M
guestfsd: main_loop: new request, len 0x3c^M
libguestfs: trace: v2v: is_file = 0

libguestfs: trace: v2v: is_file "/bin/rm" "followsymlinks:true"
guestfsd: main_loop: proc 37 (is_file) took 0.00 seconds^M
guestfsd: main_loop: new request, len 0x38^M
libguestfs: trace: v2v: is_file = 0

libguestfs: trace: v2v: is_file "/bin/sh" "followsymlinks:true"
guestfsd: main_loop: proc 37 (is_file) took 0.00 seconds^M
guestfsd: main_loop: new request, len 0x38^M
libguestfs: trace: v2v: is_file = 0

==

Finally fails with error

==
i_root = /dev/rhel/root
i_type = linux
i_distro = rhel
i_arch = unknown
i_major_version = 7
i_minor_version = 2
i_package_format = rpm
i_package_management = yum
i_product_name = Red Hat Enterprise Linux Server 7.2 (Maipo)
i_product_variant = unknown
i_uefi = false

virt-v2v: error: inspection could not detect the source guest (or physical
machine).

Inspection field 'i_arch' was 'unknown'.
==

There was a similar bug 1186935 but after this is fixed, v2v is unable to do inspection on architecture.

Version-Release number of selected component (if applicable):

libguestfs-1.32.7-3.el7.x86_64

How reproducible:

100%

Steps to Reproduce:

1. Create a RHEL 7.2 machine in VMware with /usr in a different partition.

2. Use v2v to import the virtual machine

virt-v2v -v -i ova nijin-test.ova -o rhev -of qcow2 -os 10.65.209.113:/export-domain 

3. v2v fails with error

Actual results:

v2v is failing with error 'i_arch' was 'unknown' when /usr is in different partition.

Expected results:

v2v should work when /usr is in different partition.

Additional info:

Comment 3 Richard W.M. Jones 2016-12-05 12:11:56 UTC

This is a real bug.  Inspection fails to determine the
guest architecture because the function src/inspect-fs-unix.c:
check_architecture cannot find any binary, because all of the
binaries being checked are not present on the root filesystem
(since they are only present in /usr which is mounted separately).

Comment 5 Richard W.M. Jones 2016-12-05 12:34:16 UTC

Reproducer.  This is using CentOS 7.2 guest for convenience only, but the
same problem applies to RHEL 7.2 guests with split /usr.

$ virt-builder centos-7.2 
$ mv centos-7.2.img centos-7.2.img.old
$ truncate -s 20G centos-7.2.img
$ virt-resize centos-7.2.img.old centos-7.2.img
$ rm centos-7.2.img.old
$ guestfish --rw -a centos-7.2.img <<EOF
  run
  mkmountpoint /root
  mkmountpoint /usr
  mkfs ext4 /dev/sda4
  mount /dev/sda3 /root
  mount /dev/sda4 /usr
  glob cp-a '/root/usr/*' /usr/
  glob rm-rf '/root/usr/*'
  write-append /root/etc/fstab "/dev/sda4 /usr ext4 defaults 1 2" 
EOF

Inspect the guest.  Notice that the <arch> field is missing:

$ virt-inspector centos-7.2.img | less

Try doing a V2V null conversion of the guest:

$ virt-v2v -i local centos-7.2.img -o null
[   0.0] Opening the source -i disk centos-7.2.img
[   0.0] Creating an overlay to protect the source from being modified
[   0.1] Initializing the target -o null
[   0.1] Opening the overlay
[  10.9] Inspecting the overlay
virt-v2v: error: inspection could not detect the source guest (or physical 
machine).

Assuming that you are running virt-v2v/virt-p2v on a source which is 
supported (and not, for example, a blank disk), then this should not 
happen.  You should run 'virt-v2v -v -x ... >& log' and attach the complete 
log to a new bug report (see http://libguestfs.org).

Inspection field 'i_arch' was 'unknown'.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 6 Richard W.M. Jones 2016-12-06 09:47:15 UTC

Patch series posted:

https://www.redhat.com/archives/libguestfs/2016-December/msg00013.html
https://www.redhat.com/archives/libguestfs/2016-December/msg00014.html

To reporter, please start the RHEL 7.3 z-stream process for this bug.

Comment 7 Pino Toscano 2016-12-07 14:02:36 UTC

Fixed with
https://github.com/libguestfs/libguestfs/commit/dad35e55fac526dbcd50964500f2d5047ae011af (needed bugfix for the provided OVA)
https://github.com/libguestfs/libguestfs/commit/1d86b3768956f818262a5189bb1ba996dedbf531

Comment 17 Richard W.M. Jones 2016-12-16 18:11:29 UTC

Here are some test packages which contain the RHEL 7.3 package
plus 4 patches on top required to fix this bug:

  http://oirase.annexia.org/tmp/bz1401474-inspection/

I have tested them locally, but not in great detail.  They are not
supported and not signed.  We'll try to get this fix into 7.3 z-stream,
but that's not likely to happen until January.

Comment 19 Richard W.M. Jones 2016-12-16 19:23:46 UTC

Download all the *.x86_64.rpm and *.noarch.rpm files from that
link to an empty temporary directory on the conversion server.
Then run:

  yum localupdate *.rpm

Yum will update the packages that need to be updated.

Comment 20 kuwei@redhat.com 2016-12-20 04:19:44 UTC

I can reproduce this issue with builds :
virt-v2v-1.32.7-3.el7.x86_64
libguestfs-1.32.7-3.el7.x86_64

Reproduce steps:

1：Create a RHEL 7.2 guest in ESX with /usr in a different partition.When install a new guest , one step is to configure system partitions. Click on "Installation Destination", then click "+" to set. what I setting :

/ 8G -Standard Partition
/usr 5G -Standard Partition
swap 2G

then click "Done" .
2:Select the guest which you want to convert
3:Select "File >Export > Deploy OVF Template " in Menu Bar,then select Format "Single file(OVA)",click "OK".
4:After exporting OVA file ,then send the file to v2v conversion server.
5:Use v2v to convert the OVA file
[root@localhost wk]# virt-v2v -i ova rhel7.2-usr-different-partition.ova -o null -of raw
[ 0.0] Opening the source -i ova rhel7.2-usr-different-partition.ova
tar: rhel7.2-usr-different-partition.ovf: time stamp 2016-12-19 23:43:12 is 25247.198693706 s in the future
tar: rhel7.2-usr-different-partition-disk1.vmdk: time stamp 2016-12-19 23:37:57 is 24927.180076661 s in the future
tar: rhel7.2-usr-different-partition-file1.iso: time stamp 2016-12-19 23:43:12 is 25188.542060509 s in the future
tar: rhel7.2-usr-different-partition.mf: time stamp 2016-12-19 23:43:12 is 25188.291157282 s in the future
[ 59.2] Creating an overlay to protect the source from being modified
[ 62.3] Initializing the target -o null
[ 62.3] Opening the overlay
[ 69.9] Inspecting the overlay
virt-v2v: error: inspection could not detect the source guest (or physical
machine).

Assuming that you are running virt-v2v/virt-p2v on a source which is
supported (and not, for example, a blank disk), then this should not
happen. You should run 'virt-v2v -v -x ... >& log' and attach the complete
log to a new bug report (see http://libguestfs.org).

Inspection field 'i_arch' was 'unknown'.

If reporting bugs, run virt-v2v with debugging enabled and include the
complete output:

virt-v2v -v -x [...]

So v2v convert vm rhel7.2 is failing with error 'i_arch' was 'unknown' when /usr is in different partition.

Comment 24 mxie@redhat.com 2017-03-08 05:09:36 UTC

Verify the bug with builds:
virt-v2v-1.36.1-1.el7.x86_64
libguestfs-1.36.1-1.el7.x86_64
libvirt-3.1.0-1.el7.x86_64
qemu-kvm-rhev-2.8.0-5.el7.x86_64

Steps:
1.Prepare a linux guest whose /usr is a independent partition on vmware host
2.Export this guest to ova file at vcenter client and then copy to virt-v2v conversion server
3.Using virt-v2v convert the ova file and conversion could be finished without error
]# virt-v2v -i ova rhel7.2usr-seperate -on usr-seperate -of raw
[   0.0] Opening the source -i ova rhel7.2usr-seperate
[  80.9] Creating an overlay to protect the source from being modified
[  81.7] Initializing the target -o libvirt -os default
[  81.9] Opening the overlay
[ 130.5] Inspecting the overlay
[ 147.6] Checking for sufficient free disk space in the guest
[ 147.6] Estimating space required on target for each disk
[ 147.6] Converting Red Hat Enterprise Linux Server 7.2 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 192.8] Mapping filesystem data to avoid copying unused and blank areas
[ 193.9] Closing the overlay
[ 194.1] Checking if the guest needs BIOS or UEFI to boot
[ 194.1] Assigning disks to buses
[ 194.1] Copying disk 1/1 to /var/lib/libvirt/images/usr-seperate-sda (raw)
    (100.00/100%)
[ 242.2] Creating output metadata
Pool default refreshed

Domain usr-seperate defined from /tmp/v2vlibvirt39955e.xml

[ 247.1] Finishing off

4.Power on the guest and checkpoints of guest are passed


According to above verify result, the bug has been fixed

So move this bug from ON_QA to VERIFIED

Comment 25 errata-xmlrpc 2017-08-01 22:11:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2023