Bug 509135

Summary: virt-manager not transmitting a good uri to libvirt on migration
Product: Red Hat Enterprise Linux 5 Reporter: Veaceslav Falico <vfalico>
Component: virt-managerAssignee: Cole Robinson <crobinso>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.4CC: adam.vinsh, adaora.onyia, alex_williamson, berrange, bstein, cward, doug.chapman, jjarvis, jkachuck, linda.knippers, ohudlick, peterm, qzhang, rick.hester, sghosh, shengliang.lv, tao, xen-maint, yoyzhang
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 09:42:45 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Fix KVM migration via virt-manager
none
newdesthost.log.tar
none
newvirsh.log none

Description Veaceslav Falico 2009-07-01 13:58:30 UTC
When migrating through virt-manager, we get a 'libvirtError: invalid argument in only tcp URIs are supported for KVM migrations' error.

Steps to reproduce:
1) Set up 2 connections in virt-manager (doesn't matter to what boxes, may be the same box)
2) Right-click on a vm, select 'Migrate' and select the second connection.
3) After clicking 'Yes', we get an error 'libvirtError: invalid argument in only tcp URIs are supported for KVM migrations'.


AFAICS, the error in uri starts from domain.py:
1374     def migrate(self, dictcon):
1375         flags = 0
1376         if self.lastStatus == libvirt.VIR_DOMAIN_RUNNING:
1377             flags = libvirt.VIR_MIGRATE_LIVE
1378         self.vm.migrate(self.connection.vmm, flags, None, dictcon.get_short_hostname(), 0)
line 1378 - the uri parameter should be either NULL for local or a 'tcp:host:port' for remote. I've tried to find a suitable function, however none found. If we make '"tcp:" + dictcon.get_short_hostname() + ":portnumber"' things work.

Comment 1 Daniel Berrangé 2009-07-01 14:02:38 UTC
virt-manager should not be giving any URI to migrate() at all. libvirt correctly fills this in 99% of the time.

You only need to pass URI, if the host has multiple NICs, and you want migration togo over a non-default one.

Comment 2 Issue Tracker 2009-07-01 17:12:58 UTC
Event posted on 07-01-2009 01:12pm EDT by jkachuck

Redhat, Thanks for the update. While trying migration using virt manager,
the migration has not finished even after 15 minutes.

We already mentioned that a)Using virsh
The guest was able to get migrated using virsh through qemu+ssh and
qemu+tcp, But while migrating using virsh, we observed but most of the
times the migration was kind of halted and was never ending.So,we had to
kill the process and get it started again.


This event sent from IssueTracker by jkachuck 
 issue 305833

Comment 3 Cole Robinson 2009-07-06 16:39:37 UTC
Created attachment 350640 [details]
Fix KVM migration via virt-manager

This patch maintains the existing behavior for xen, since it works. Calling 'migrate' on xen with the same arguments that work for KVM causes the call to bail out immediately. Haven't poked into libvirt to determine why yet.

Either way, the patch makes both kvm and xen migrations appear to succeed.

Comment 4 Cole Robinson 2009-07-06 16:42:15 UTC
FYI, migration is new behavior for virt-manager in 5.4, so this change can't really regress (it also isn't a regression). That said, these changes are low risk, local to the migration code only, and make KVM migration via virt-manager actually work.

I think this should be marked as an exception.

Comment 5 Joseph Kachuck 2009-07-07 14:30:59 UTC
Update from client:

  While using virsh for migrating the guest, it gets kind of halted most of the times and becomes never ending.This is not seen in the case of "virt-manager".In virt-manager,the migration doesn't start at all. Attaching the tcpdump for the destination host while the guest was getting migrated and halted.Strangely, the tcpdump for the source host was empty.
Setup was fine and selinux was also disabled.

Thanks,
Santwana


After applying the patch,I tried to migrate the guest using the virt-manager.But when triggering the migration using virt-manager,I noticed that "virt-manager" hangs and inturn the guest console also hangs.In the destination host,opening virt-manager resulted in the similar behavior of halted.We have to force-quit the application to get rid of it.
Similar behavior was noticed using virsh, while migrating the guest,as mentioned earlier.

Thanks,
Santwana

Comment 6 Joseph Kachuck 2009-07-07 14:31:33 UTC
Created attachment 350808 [details]
newdesthost.log.tar

Comment 7 Joseph Kachuck 2009-07-07 14:32:03 UTC
Created attachment 350809 [details]
newvirsh.log

Comment 9 Cole Robinson 2009-07-07 18:28:33 UTC
Built into virt-manager-0.6.1-6.el5. Setting to MODIFIED.

Joe, if you can reproduce migration issues against virsh, file a bug against libvirt. This bug is only tracking a specific issue with virt-manager KVM migration, namely that it would bail out immediately with a libvirtError.

Comment 12 Issue Tracker 2009-07-08 13:31:49 UTC
Event posted on 07-08-2009 09:31am EDT by jkachuck

From Client:
Hi,

   I have created an attachment for the tcpdump transfer for the
destination host, while the guest was migrated from source using
"virt-manager". As soon as the migration was triggered using
"virt-manager",virt-manger stopped responding. There was no data in the
tcpdump for the source host.

Thanks,
Santwana

Internal Status set to 'Waiting on Engineering'

This event sent from IssueTracker by jkachuck 
 issue 305833
it_file 235661

Comment 13 Cole Robinson 2009-07-08 16:08:10 UTC
*** Bug 510291 has been marked as a duplicate of this bug. ***

Comment 14 Issue Tracker 2009-07-13 14:16:21 UTC
Event posted on 07-13-2009 10:16am EDT by jkachuck

Hello Redhat,

I was able to migrate the guest in RHEL5.4-Snap1 release after applying
the patch.I had added the connection to the destination host as
"qemu+ssh". Migration was successful for first 1-2 iterations between
the 2 hosts. But later, the same issue of the virt-manager getting hanged
is noticed again.

The migration is not consistent using virt-manager. During the 3rd or 4th
iteration,the virt-manager in both the hosts hangs. The command "virsh"
also hangs. The issue doesnot get resolved until the host is
rebooted.Overall, the behavior of "virt-manager" for migration is still
not consistent.

Thanks,
Santwana 


This event sent from IssueTracker by jkachuck 
 issue 305833

Comment 18 Cole Robinson 2009-07-17 13:55:51 UTC
Then that is a separate issue. This bug is not for 'virt-manager migration doesn't work', this bug is for 'virt-manager sending incorrect paramaters to migrate command'. The best way for us to solve this different bug is to file a separate report so that everyone is clear it is an independent and more involved issue.

In the new bug, please specify:

- if this is reproducable using only virsh migrate
- /var/log/libvirt/qemu/<vmname>.log on both the sending and receiving hosts
- XML config of the VM being migrated.
- hypervisor (KVM is mentioned above, but just to clarify).

If this is also reproducible against 'virsh', please file the bug against libvirt.

Comment 19 Issue Tracker 2009-07-17 15:46:53 UTC
Event posted on 07-17-2009 10:29am EDT by Glen Johnson

------- Comment From tpnoonan.com 2009-07-17 10:15 EDT-------
IBM does have a requirement that for libvirt and virt-manager migration
working in RHEL5.4. There is a suggestion that there may be an issue
between the he version of libvirt and KVM.

------- Comment From mohan.com 2009-07-17 10:17 EDT-------
Redhat,

We are referring to the LTC bugzilla 53835 (Redhat bugzilla 305833) Unable
to
migrate a VM using virt-manager in RHEL5.4-pre Alpha release. We already
opened
a bugzilla to track virsh migration problems.


This event sent from IssueTracker by jkachuck 
 issue 305833

Comment 21 Cole Robinson 2009-07-21 14:00:57 UTC
Anyone who can reproduce: please provide the info in comment #18 so we can debug this appropriately.

Comment 22 Veaceslav Falico 2009-07-22 10:00:38 UTC
I'll open a new BZ for #12, because it is a different issue, not related to uri parameter transmitted to libvirt.

Comment 23 zhanghaiyan 2009-07-22 10:16:23 UTC
Verified migrate via virt-manager, can migrate successfully without no error 

Version:
- xen-3.0.3-90.el5
- libvirt-0.6.3-15.el5
- kvm-83-92.el5
- RHEL5.4 (2.6.18-158.el5)
- virt-manager-0.6.1-7.el5


Steps to reproduce:
1) Set up 1 connections in virt-manager 
2) Right-click on a vm, select 'Migrate' and select the connection.
3) After clicking 'Yes', migrate successfully

Comment 24 zhanghaiyan 2009-07-22 10:19:12 UTC
(In reply to comment #18)
 Verified live migration kvm guest with virsh.
Can migrate guest, and migrate guest back successfully.

Version:
- xen-3.0.3-90.el5
- libvirt-0.6.3-15.el5
- kvm-83-92.el5
- RHEL5.4 (2.6.18-158.el5)

Steps:
On both source and target machine
1. # iptables -F
2. # setenforce 0
3. # mount 10.66.71.226:/vol/kvm1/auto/images_good /var/lib/libvirt/images

On source machine
4. # wget http://10.66.70.202/~yoyzhang/libvirt/test_file/test1-kvm.xml
   Modify test1-kvm.html as
   <source file="/var/lib/libvirt/images/RHEL-Server-5.4-64.qcow2"/>
5. # virsh define test1-kvm.xml
6  # virsh start test1
7  # virsh list --all
   ID   Name   State
--------------------------
   10  test1    running 
8. # virsh migrate --live test1 qemu+ssh://10.66.70.188/system
   root.70.188's password:


Actual result:
On target machine
(Migrate success in about 5 seconds)
9. # virsh list --all
      ID   Name   State
--------------------------
   1  test1    running 

And test1 works well on target machine
Additional info: 
Can migrate test1 back from target machine to source machine, and test1 works
well.

Comment 25 Cole Robinson 2009-07-22 19:21:43 UTC
QA has indicated this issue is fixed in commeent #23 and comment #24, so moving to VERIFIED

Comment 26 Issue Tracker 2009-07-24 13:24:48 UTC
Event posted on 07-23-2009 10:07am EDT by Glen Johnson

------- Comment From markwiz.com 2009-07-23 10:01 EDT-------
Red Hat lists versions of KVM related rpms which they have successfully
tested migration with.
Version:
- xen-3.0.3-90.el5
- libvirt-0.6.3-15.el5
- kvm-83-92.el5
- RHEL5.4 (2.6.18-158.el5)

Snap3 has kvm-83-90.e15 instead of kvm-83-92.e15.

Santwana will try the version that came with snap3.

If Snap3 still does not work, I have downloaded the snap4 versions and
they
are on GSA under ltc_test/RHEl5.4.

Internal Status set to 'Waiting on Support'
Status set to: Waiting on Tech

This event sent from IssueTracker by jkachuck 
 issue 305833

Comment 27 Issue Tracker 2009-07-24 13:24:53 UTC
Event posted on 07-24-2009 07:03am EDT by Glen Johnson

<cde:attachment>
Comment on attachment: virt-manager.log

------- Comment on attachment From santwana.samantray.com
2009-07-24 06:49 EDT-------


Hello Redhat,

            I verified this issue on RHEL5.4-Snap3. Below are the
versions:
Version:
xen-3.0.3-90.el5
libvirt-0.6.3-15.el5
kvm-83-90.el5
RHEL5.4(2.6.18-158.el5)

         First few iterations of the migration was successful using
"remote tunnel over ssh" as the connection type to the
destination.Later, in subsequent tries, the virt-manager was unresponsive
and was struck, and migration failed.Attaching virt-manager.log for
reference.

Thanks,
Santwana
</cde:attachment>


This event sent from IssueTracker by jkachuck 
 issue 305833

Comment 28 John Jarvis 2009-07-24 13:31:08 UTC
Actually on a more careful re-reading of comment 27 you will notice the tester is stating that the problem still exists.  Setting to FAILS_QA to trigger review.

Comment 29 Daniel Berrangé 2009-07-24 13:41:00 UTC
The comment about migration getting stuck / failing, is unrelated to this bug, which is specifically tracking the problem of virt-manager not providing the correct URI. Since migration worked even 1 time, then virt-manager is clearly providing the correct URI, and thus this bug should be left in the  VERIFIED state.

Please file a separate bug for any additional, unrelated issues with migration failing.

Comment 30 Cole Robinson 2009-07-24 14:50:44 UTC
I believe the 'migration hangs' issue is being tracked now in bug 512975

Comment 33 errata-xmlrpc 2009-09-02 09:42:45 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1285.html

Comment 34 Shengliang Lv 2009-09-02 10:22:47 UTC
Reopen this issue cause the ERRATA link isn't access.