Bug 1142008

Summary: Conversion guest cannot been seen on rhevm server
Product: Red Hat Enterprise Linux 7 Reporter: zhoujunqin <juzhou>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: codong, dyuan, juzhou, mbooth, mzhan, ptoscano, rjones, tzheng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.27.48-1.1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 13:44:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1143887    
Bug Blocks:    
Attachments:
Description Flags
Screenshot-1 on rhevm server
none
log-1 got from rhevm server:/var/log/ovirt-engine/engine.log
none
Screenshot-2
none
Screenshot-3
none
Screenshot-4
none
Attched the log file to convert to rhev none

Description zhoujunqin 2014-09-16 02:07:45 UTC
Description of problem:
Conversion guest cannot been seen on rhevm

Version-Release number of selected component (if applicable):

virt-v2v-1.27.43-1.1.el7.x86_64
libguestfs-1.27.43-1.1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a shutdown guest.
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     rhel6                          shut off

2. Use virt-v2v to convert the guest "rhel6" to rhevm server
# virt-v2v -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export --network rhevm rhel6
[   0.0] Opening the source -i libvirt rhel6
[   0.0] Creating an overlay to protect the source from being modified
[   0.0] Opening the overlay
[   3.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export
[   3.0] Inspecting the overlay
[  13.0] Checking for sufficient free disk space in the guest
[  13.0] Converting Red Hat Enterprise Linux Server release 6.6 Beta (Santiago) to run on KVM
[  42.0] Mapping filesystem data to avoid copying unused and blank areas
[  42.0] Closing the overlay
[  43.0] Copying disk 1/1 to /tmp/v2v.xRvWQu/5b625e34-c943-4407-949d-9b9926bdf406/images/d689cfc3-3b55-44fb-bf01-62554224f843/1d623a53-5094-4ac1-92ae-0be9ce5f5adf (raw)
    (100.00/100%)
[ 114.0] Creating output metadata
[ 114.0] Finishing off

3. Login in rhevm server using Administrator Portal, try to import the domain converted(rhel6), details see screenshot-1 and rhevm-log-1.

Actual results:
Conversion guest rhel6 cannot been seen on rhevm server.

Expected results:
Conversion guest rhel6 can been seen and imported successfully.

Additional info:
The guest converted from rhel6 guest can be seen and imported successfully.

Comment 1 zhoujunqin 2014-09-16 02:10:32 UTC
Created attachment 937776 [details]
Screenshot-1 on rhevm server

Comment 3 zhoujunqin 2014-09-16 02:39:32 UTC
Created attachment 937777 [details]
log-1 got from rhevm server:/var/log/ovirt-engine/engine.log

Comment 4 Richard W.M. Jones 2014-09-16 07:44:41 UTC
Just to be sure, /vol/v2v_auto/auto_export is where you have
mounted the Export Storage Domain (NOT the data domain) of the
RHEV-M installation?

I will have to boot up my RHEV-M system later today and
check that import still works ...

Comment 6 zhoujunqin 2014-09-16 11:11:22 UTC
Created attachment 937929 [details]
Screenshot-2

Comment 7 zhoujunqin 2014-09-16 11:11:59 UTC
Created attachment 937930 [details]
Screenshot-3

Comment 8 zhoujunqin 2014-09-16 11:12:33 UTC
Created attachment 937932 [details]
Screenshot-4

Comment 9 Richard W.M. Jones 2014-09-16 21:57:59 UTC
Sadly I am able to reproduce this, which must mean something has
changed in how we lay out the VM files in the RHEV storage domain.

Comment 10 Richard W.M. Jones 2014-09-17 13:02:21 UTC
There were various regressions added during code refactoring which
broke the import.  Unfortunately we don't have a way to automatically
test this.  Anyway the fixes are:

https://github.com/libguestfs/libguestfs/commit/fe36b2ce950d214aa198daf86188e8fdbe41f43f
https://github.com/libguestfs/libguestfs/commit/05f665166c05519b8b5d4cd26fbe5e912cc63ba8
https://github.com/libguestfs/libguestfs/commit/d5001304e4952450851ab429416c96cc4c8e2331

and related to this:

https://github.com/libguestfs/libguestfs/commit/dd58cde929ade2dd6bc216aac21658ed3b3f51aa

These will be added to virt-v2v >= 1.27.48.

Comment 12 tingting zheng 2014-09-18 08:03:16 UTC
Tested with:
libguestfs-1.27.48-1.1.el7.x86_64
virt-v2v-1.27.48-1.1.el7.x86_64

Steps:
1.Use virt-v2v to convert a guest to rhev.
#virt-v2v -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export --network rhevm rhel6.6-tzheng -v -x |& tee /tmp/v2v-rhevm.log

2.After conversion,the guest can be seen in auto export,however,when I try to import it,error shows in rhevm GUI:
Operation Canceled
Error while executing action:

rhel6.6-tzheng:

    Cannot import VM. VM's Image does not exist.

So I think the bug is not fixed,move back to ASSIGNED.

Comment 13 tingting zheng 2014-09-18 08:08:13 UTC
Created attachment 938788 [details]
Attched the log file to convert to rhev

Comment 14 Richard W.M. Jones 2014-09-18 08:09:45 UTC
(In reply to tingting zheng from comment #12)
> Tested with:
> libguestfs-1.27.48-1.1.el7.x86_64
> virt-v2v-1.27.48-1.1.el7.x86_64
> 
> Steps:
> 1.Use virt-v2v to convert a guest to rhev.
> #virt-v2v -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export --network rhevm
> rhel6.6-tzheng -v -x |& tee /tmp/v2v-rhevm.log
> 
> 2.After conversion,the guest can be seen in auto export,however,when I try
> to import it,error shows in rhevm GUI:
> Operation Canceled
> Error while executing action:
> 
> rhel6.6-tzheng:
> 
>     Cannot import VM. VM's Image does not exist.

I saw this error and it was caused by virt-v2v not 'chown'-ing
the files correctly (to 36.36).  However I have fixed this in .48
So perhaps the error is caused by something else.

Unfortunately the errors shown in the GUI don't tell you what's
really happening.  For that you have to look at the 'engine.log' file.

Can you log on to the RHEV-M server and find the file
/var/log/ovirt/engine.log (I'm not sure of the precise name, but
it's something like that).

Then find out the real error from that log.  If you can't find it,
then attach the whole log.

Comment 15 Richard W.M. Jones 2014-09-18 08:11:01 UTC
Actually I can see the problem.  virt-v2v printed this warning.
I wonder why?

chown -R 36.36 '/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-ccc0-4ab1-9b5f-e212fe3bbb4d' '/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-492f-bda3-62e0c68ec3ca'
chown: changing ownership of ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-ccc0-4ab1-9b5f-e212fe3bbb4d/434419ca-ccc0-4ab1-9b5f-e212fe3bbb4d.ovf’: Invalid argument
chown: changing ownership of ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-ccc0-4ab1-9b5f-e212fe3bbb4d’: Invalid argument
chown: changing ownership of ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-492f-bda3-62e0c68ec3ca/f9cd21a3-b01d-4ded-9051-e19c69cb0c9f.meta’: Invalid argument
chown: changing ownership of ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-492f-bda3-62e0c68ec3ca/f9cd21a3-b01d-4ded-9051-e19c69cb0c9f’: Invalid argument
chown: changing ownership of ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-492f-bda3-62e0c68ec3ca’: Invalid argument
virt-v2v: warning: could not chown newly created RHEV files and directories 
to 36.36. You may need to do this by hand, otherwise RHEV-M may give errors 
when trying to import this domain.

The directories (and all files inside) that have to be owned by 36.36 are:

46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-ccc0-4ab1-9b5f-e212fe3bbb4d

46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-492f-bda3-62e0c68ec3ca

Comment 16 tingting zheng 2014-09-18 08:13:27 UTC
(In reply to Richard W.M. Jones from comment #15)
> Actually I can see the problem.  virt-v2v printed this warning.
> I wonder why?
> 
> chown -R 36.36
> '/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-
> ccc0-4ab1-9b5f-e212fe3bbb4d'
> '/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-
> 492f-bda3-62e0c68ec3ca'
> chown: changing ownership of
> ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-
> ccc0-4ab1-9b5f-e212fe3bbb4d/434419ca-ccc0-4ab1-9b5f-e212fe3bbb4d.ovf’:
> Invalid argument
> chown: changing ownership of
> ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-
> ccc0-4ab1-9b5f-e212fe3bbb4d’: Invalid argument
> chown: changing ownership of
> ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-
> 492f-bda3-62e0c68ec3ca/f9cd21a3-b01d-4ded-9051-e19c69cb0c9f.meta’: Invalid
> argument
> chown: changing ownership of
> ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-
> 492f-bda3-62e0c68ec3ca/f9cd21a3-b01d-4ded-9051-e19c69cb0c9f’: Invalid
> argument
> chown: changing ownership of
> ‘/tmp/v2v.u48xag/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-
> 492f-bda3-62e0c68ec3ca’: Invalid argument
> virt-v2v: warning: could not chown newly created RHEV files and directories 
> to 36.36. You may need to do this by hand, otherwise RHEV-M may give errors 
> when trying to import this domain.
> 
> The directories (and all files inside) that have to be owned by 36.36 are:
> 
> 46adae8a-63c1-40f8-b25a-f02deb1a5160/master/vms/434419ca-ccc0-4ab1-9b5f-
> e212fe3bbb4d
> 
> 46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-492f-bda3-
> 62e0c68ec3ca

For these warning info,I filed bug:Bug 1143887

Comment 17 tingting zheng 2014-09-18 08:22:37 UTC
(In reply to Richard W.M. Jones from comment #14)
> (In reply to tingting zheng from comment #12)
> > Tested with:
> > libguestfs-1.27.48-1.1.el7.x86_64
> > virt-v2v-1.27.48-1.1.el7.x86_64
> > 
> > Steps:
> > 1.Use virt-v2v to convert a guest to rhev.
> > #virt-v2v -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export --network rhevm
> > rhel6.6-tzheng -v -x |& tee /tmp/v2v-rhevm.log
> > 
> > 2.After conversion,the guest can be seen in auto export,however,when I try
> > to import it,error shows in rhevm GUI:
> > Operation Canceled
> > Error while executing action:
> > 
> > rhel6.6-tzheng:
> > 
> >     Cannot import VM. VM's Image does not exist.
> 
> I saw this error and it was caused by virt-v2v not 'chown'-ing
> the files correctly (to 36.36).  However I have fixed this in .48
> So perhaps the error is caused by something else.
> 
> Unfortunately the errors shown in the GUI don't tell you what's
> really happening.  For that you have to look at the 'engine.log' file.
> 
> Can you log on to the RHEV-M server and find the file
> /var/log/ovirt/engine.log (I'm not sure of the precise name, but
> it's something like that).
> 
> Then find out the real error from that log.  If you can't find it,
> then attach the whole log.

I didn't find any error info from engine.log when I try to import the guest.
I find error info from rhevh,vdsm.log:
age.46adae8a-63c1-40f8-b25a-f02deb1a5160`ReqID=`b470e87e-3367-43b1-897c-0f1a573f0d18`::Granted request
Thread-21513::DEBUG::2014-09-18 08:17:28,858::task::827::TaskManager.Task::(resourceAcquired) Task=`8b980ed5-39d0-4e31-b813-b5c39042ae15`::_resourcesAcquired: Storage.46adae8a-63c1-40f8-b25a-f02deb1a5160 (shared)
Thread-21513::DEBUG::2014-09-18 08:17:28,859::task::990::TaskManager.Task::(_decref) Task=`8b980ed5-39d0-4e31-b813-b5c39042ae15`::ref 1 aborting False
Thread-21513::ERROR::2014-09-18 08:17:28,860::task::866::TaskManager.Task::(_setError) Task=`8b980ed5-39d0-4e31-b813-b5c39042ae15`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
  File "/usr/share/vdsm/storage/hsm.py", line 3093, in getVolumeInfo
  File "/usr/share/vdsm/storage/sd.py", line 399, in produceVolume
  File "/usr/share/vdsm/storage/fileVolume.py", line 56, in __init__
  File "/usr/share/vdsm/storage/volume.py", line 140, in __init__
  File "/usr/share/vdsm/storage/volume.py", line 152, in validate
  File "/usr/share/vdsm/storage/fileVolume.py", line 490, in validateImagePath
ImagePathError: Image path does not exist or cannot be accessed/created: ('/rhev/data-center/mnt/10.66.90.115:_vol_v2v__auto_auto__export/46adae8a-63c1-40f8-b25a-f02deb1a5160/images/2553e426-cf79-492f-bda3-62e0c68ec3ca',)
Thread-21513::DEBUG::2014-09-18 08:17:28,861::task::885::TaskManager.Task::(_run) Task=`8b980ed5-39d0-4e31-b813-b5c39042ae15`::Task._run: 8b980ed5-39d0-4e31-b813-b5c39042ae15 ('46adae8a-63c1-40f8-b25a-f02deb1a5160', '00000002-0002-0002-0002-0000000002c4', '2553e426-cf79-492f-bda3-62e0c68ec3ca', 'f9cd21a3-b01d-4ded-9051-e19c69cb0c9f') {} failed - stopping task
Thread-21513::DEBUG::2014-09-18 08:17:28,861::task::1211::TaskManager.Task::(stop) Task=`8b980ed5-39d0-4e31-b813-b5c39042ae15`::stopping in state preparing (force False)

Comment 18 tingting zheng 2014-09-18 08:35:57 UTC
It seems the privilege of the file is not correct:
on rhevh host:
# pwd
/rhev/data-center/mnt/10.66.90.115:_vol_v2v__auto_auto__export/46adae8a-63c1-40f8-b25a-f02deb1a5160
[root@dell-op780-04 46adae8a-63c1-40f8-b25a-f02deb1a5160]# ll -lsh --->the files under this directory is vdsm.kvm.
total 12K
4.0K drwxr-xr-x. 2 vdsm kvm 4.0K 2014-09-17 22:53 dom_md
4.0K drwxr-xr-x. 4 vdsm kvm 4.0K 2014-09-18 00:22 images
4.0K drwxr-xr-x. 4 vdsm kvm 4.0K 2014-09-17 22:53 master

# cd images   ----> the files under this directory is root.root.
[root@dell-op780-04 images]# ls
2553e426-cf79-492f-bda3-62e0c68ec3ca  _remoV2IgiD
[root@dell-op780-04 images]# ll -lsh
total 8.0K
4.0K drwxr-xr-x. 2 root root 4.0K 2014-09-18 01:01 2553e426-cf79-492f-bda3-62e0c68ec3ca
4.0K drwxr-xr-x. 2 root root 4.0K 2014-09-17 22:56 _remoV2IgiD
[root@dell-op780-04 images]# cd 2553e426-cf79-492f-bda3-62e0c68ec3ca/
[root@dell-op780-04 2553e426-cf79-492f-bda3-62e0c68ec3ca]# ll -lsh
total 3.1G
3.1G -rw-r--r--. 1 root root 10G 2014-09-18 00:00 f9cd21a3-b01d-4ded-9051-e19c69cb0c9f
4.0K -rw-r--r--. 1 root root 296 2014-09-17 23:57 f9cd21a3-b01d-4ded-9051-e19c69cb0c9f.meta

Comment 19 tingting zheng 2014-10-31 03:39:34 UTC
Tested with latest virt-v2v version,guests can be showed in rhev GUI after conversion,as the warning issue during conversion is tracked by bug 1143887,so would you pls move the bug to ON_QA,then we can verify it,thanks.

Comment 21 tingting zheng 2014-10-31 08:10:17 UTC
Tested with:
virt-v2v-1.28.1-1.6.el7.x86_64
libguestfs-1.28.1-1.6.el7.x86_64

Use virt-v2v to convert a guest to rhev,guest can be showed in rhevm GUI,so move the bug to VERIFIED.

Comment 23 errata-xmlrpc 2015-03-05 13:44:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0303.html