Bug 860573 - Live migration from rhel6.3 release version to rhel6.4 newest version with 501MB memory in guest will fail
Live migration from rhel6.3 release version to rhel6.4 newest version with 50...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.4
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Markus Armbruster
Virtualization Bugs
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-26 04:12 EDT by EricLee
Modified: 2013-02-21 02:40 EST (History)
22 users (show)

See Also:
Fixed In Version: qemu-kvm-0.12.1.2-2.335.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-21 02:40:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
source of rhel6u3 host libvirtd log (700.58 KB, text/plain)
2012-10-07 22:55 EDT, EricLee
no flags Details
destination of rhel6u4 host libvirtd log (324.54 KB, text/plain)
2012-10-07 22:56 EDT, EricLee
no flags Details

  None (edit)
Description EricLee 2012-09-26 04:12:48 EDT
Description of problem:
Do live migration from rhel6.3 release version to rhel6.4 newest version with less memory in guest will get unexpectedly failed

Version-Release number of selected component (if applicable):
6.4 latest packages:
# rpm -qa libvirt qemu-kvm-rhev; uname -r
libvirt-0.10.2-1.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.313.el6.x86_64
2.6.32-306.el6.x86_64
6.3 release packages:
# rpm -qa libvirt qemu-kvm-rhev; uname -r
qemu-kvm-rhev-0.12.1.2-2.295.el6.x86_64
libvirt-0.9.10-21.el6.x86_64
2.6.32-279.el6.x86_64

How reproducible:
100%

Steps:
1. Prepare 2 hosts, host 1 is rhel6.3 latest release version, host 2  is newest rhel6.4 version

2. Prepare a nfs which is mounted on both hosts, and setting the virt_use_nfs boolean on both sides

  # setsebool virt_use_nfs 1

   and close the iptable on both sides

   # iptables -F

3. Prepare a guest locate on host 1 and start, check the qemu-kvm command

the memory xml part use:
  <memory unit='KiB'>513024</memory>
  <currentMemory unit='KiB'>513024</currentMemory>

and then start it.

4. Do live migration from host 1 to host 2

# virsh migrate --live guest qemu+ssh://{host 2 ip}/system
error: operation failed: migration job: unexpectedly failed

Actual result
As steps

Expect result
No error and succeed.

Additional info
I have tried to find the reason for it.
I found when I define guest with:
  <memory unit='KiB'>513024</memory>
  <currentMemory unit='KiB'>513024</currentMemory>
in rhel6u4 with latest libvirt packages host, define succeed, and then start it will cause "currentMemory" increasing to 514048(KiB) automatically and start successfully, which is not expected.
And if I migrate that guest to another host(both 6u4 and 6u3), will get error:
error: XML error: current memory '514048k' exceeds maximum '513024k'
Comment 2 Michal Privoznik 2012-10-01 02:59:49 EDT
Eric,

can you please attach debug logs from both source and destination?
To set it up, edit /etc/libvirt/libvirtd.conf and set these variables:
  log_level=1
  log_outputs="1:file:/var/log/libvirtd.log"
on both source and destination. It will produce /var/log/libvirtd.log which you want to attach. Thanks.
Comment 3 EricLee 2012-10-07 22:53:39 EDT
(In reply to comment #2)
> Eric,
> 
> can you please attach debug logs from both source and destination?
> To set it up, edit /etc/libvirt/libvirtd.conf and set these variables:
>   log_level=1
>   log_outputs="1:file:/var/log/libvirtd.log"
> on both source and destination. It will produce /var/log/libvirtd.log which
> you want to attach. Thanks.

Hi Michal,

Sorry for delaying to reply, we were in a holiday for one week.

Please see the next two comments.

Thanks,
EricLee
Comment 4 EricLee 2012-10-07 22:55:13 EDT
Created attachment 623239 [details]
source of rhel6u3 host libvirtd log
Comment 5 EricLee 2012-10-07 22:56:15 EDT
Created attachment 623240 [details]
destination of rhel6u4 host libvirtd log
Comment 6 Michal Privoznik 2012-10-09 11:35:16 EDT
Eric,

I've managed to reproduce this issue and I can see the error message in qemu log (/var/log/libvirt/qemu/$domain.log on the destination) saying:

qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed
2012-10-09 15:32:25.928+0000: shutting down


Is this your case as well? This would signal that qemu cannot migrate between reported versions. Otherwise I have not spotted anything wrong in the logs.
Comment 7 EricLee 2012-10-09 22:13:02 EDT
(In reply to comment #6)
> Eric,
> 
> I've managed to reproduce this issue and I can see the error message in qemu
> log (/var/log/libvirt/qemu/$domain.log on the destination) saying:
> 
> qemu: warning: error while loading state for instance 0x0 of device 'ram'
> load of migration failed
> 2012-10-09 15:32:25.928+0000: shutting down
> 
> 
> Is this your case as well? This would signal that qemu cannot migrate
> between reported versions. Otherwise I have not spotted anything wrong in
> the logs.

Yeah, I got the same error in /var/log/libvirt/qemu/$domain.log on destination host.

So I think it is due to the unexpected memory value when start the guest, which I have said in Additional info in comment #0.
Comment 8 Michal Privoznik 2012-10-12 05:43:01 EDT
Eric, I think this is actually qemu-kvm bug.

I've downgraded libvirt on dst to minimize impact of libvirt (qemu cmd line stays the same) and still wasn't able to migrate. I think we need somebody from qemu to take a look. Therefore I am switching component to qemu-kvm.

Here's the cmd line:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 501 -smp 2,sockets=2,cores=1,threads=1 -name fdr -uuid 6d74cdcb-b28a-36f6-de2d-59b2f3419eb6 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fdr.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/masina_nfs/fdr.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:4d:af:4f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -incoming tcp:0.0.0.0:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
Comment 9 Paolo Bonzini 2012-10-16 09:42:29 EDT
Eric, can you please try these tests?

1) source and dest being both 6.3, with -M rhel6.3.0

2) source and dest being both 6.4, with -M rhel6.3.0

3) source and dest being both 6.4, with -M rhel6.4.0
Comment 10 EricLee 2012-10-16 22:48:43 EDT
(In reply to comment #9)
> Eric, can you please try these tests?
> 
> 1) source and dest being both 6.3, with -M rhel6.3.0

Migration succeed without error.

In this scene, guest memory do not automatically change to 514048k.

> 
> 2) source and dest being both 6.4, with -M rhel6.3.0

Failed with error:
error: XML error: current memory '514048k' exceeds maximum '513024k'

In this scene, guest memory will automatically change to 514048k, which is not expected.

> 
> 3) source and dest being both 6.4, with -M rhel6.4.0

Migration succeed without error.

In this scene, guest memory do not automatically change to 514048k.

And you can also refer to the "Additional info" of description in the bug.
Comment 11 Markus Armbruster 2012-10-17 10:12:52 EDT
Juan debugged this.  Culprit is commit b7497a17 "vl: Round argument of -m up to multiple of 2MiB instead of 8KiB".  Old qemu interprets "-m 501" as 501MiB, new qemu as 502MiB.  Migration fails unless RAM size matches exactly.

Proposed fix: round up to multiple to page size (regression-proof, because partial pages never worked), and enforce minimum 2MiB (okay, because no supported guest will boot with less than that.
Comment 19 langfang 2013-01-05 04:37:19 EST
Reproduce this bug as follow version:
Host A:rhel6.3.z
# uname -r
2.6.32-279.11.1.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.295.el6.x86_64

Host B:rhel6.4-latest
# uname -r
2.6.32-348.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.334.el6.x86_64

Steps:
1.Boot guest 
Host A:
 /usr/libexec/qemu-kvm -M rhel6.3.0 -cpu Conroe -enable-kvm -m 501M -smp 1,sockets=1,cores=1,threads=1 -name SF-VM1 -uuid c51aafca-18b1-4f65-aa09-38632e9d027f -monitor stdio -rtc base=localtime,clock=host,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/mnt/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=60137e34-ef49-4bed-8a67-7f90d1425414,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,script=/etc/qemu-ifup,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:b6:62:8f,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/tmp/qzhang,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -device usb-tablet,id=input0 -vnc :10 -k en-us -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

Host B:
....-incoming tcp:0:5999
2.Migration
Src host:
(qemu)migrate -d tcp:10.66.4.195:5999

Results:
(qemu) qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed


Verify this bug as follow version:

Host A:rhel6.3.z
# uname -r
2.6.32-279.11.1.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.295.el6.x86_64

Host B:rhel6.4-latest
# uname -r
2.6.32-348.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.348.el6.x86_64

Guest:
# uname -r
2.6.32-343.el6.x86_64

Steps as same as Reproduce.

Results:
Not show any error,migrate successfully.

Addtional info:
Also test other five senarios:

Senario 1) source and dest being both 6.4, with -M rhel6.4.0
Version:
Host A AND Host B
[root@amd-4450b-4-2 home]# uname -r
2.6.32-351.el6.x86_64
[root@amd-4450b-4-2 home]# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.348.el6.x86_64

Guest:
[root@amd-4450b-4-2 home]# uname -r
2.6.32-343.el6.x86_64

Steps as same as reproduce.

Results:
Not show any error,migrate successfully.

Senario 2) source and dest being both 6.4, with -M rhel6.3.0

Test version as same as Senario 1.

Results:
Not show any error,migrate successfully.

Senario 3) source and dest being both 6.3, with -M rhel6.3.0

Results:
Not show any error,migrate successfully.

Senario 4) Live migration from rhel6.2 release version to rhel6.4 newest version,with -M rhel6.2.0

Not show any error,migrate successfully.
Senario 5) Live migration from rhel6.1 release version to rhel6.4 newest version,with -M rhel6.1.0

To avoid bug 824399,Use ide disk to test .Not show any error,migrate successfully.


According to above test ,this bug fixed.
Comment 21 errata-xmlrpc 2013-02-21 02:40:02 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0527.html

Note You need to log in before you can comment on or make changes to this bug.