Bug 504264

Created attachment 346617 [details]
strace result

Other OS OK?

can migrate on 5.3 host

Please pin the guest to a specific CPU (using taskset) and run kvmtrace for 1 second.

Please re-test using kvm-83-72

the same result:
1. command used:
qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack -m 1G -uuid `uuidgen` -net nic,model=e1000,macaddr=00:1a:4a:16:66:86,vlan=0 -net tap,vlan=0,script=/etc/qemu-ifup -drive file=5.4s-64.qcow2,if=ide -boot c -vnc :10
2. top
5547 root      15   0 1223m 1.0g 2488 S 100.9  8.8   0:35.12 qemu-kvm
    1 root      15   0 10348  692  584 S  0.0  0.0   0:01.12 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/1
    6 root      34  19     0    0    0 S  0.0  0.0   0:01.30 ksoftirqd/1
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/2
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/2
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/3
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/3
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3
   14 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/4
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/4

3. kvm_stat:
 efer_reload                  0       0
 exits                 11948201  158472
 fpu_reload                  46       0
 halt_exits                6092       0
 halt_wakeup                 34       0
 host_state_reload     10836795  144278
 hypercalls                   0       0
 insn_emulation         3312346   43889
 insn_emulation_fail          0       0
 invlpg                       0       0
 io_exits              10829757  144276
 irq_exits                 2432      31
 irq_injections          519128    6655
 irq_window              493493    6510
 kvm_request_irq              0       0
 largepages                   0       0
 mmio_exits                 450       0
 mmu_cache_miss             484       0
 mmu_flooded                  0       0
 mmu_pde_zapped               0       0
 mmu_pte_updated              0       0

4. strace: attached
5. kvmtrace: attached
6. kvm version:
kvm-83-73.el5

Created attachment 347008 [details]
strace new

Created attachment 347009 [details]
kvmtrace, can not read derictory, need to analyze first. I failed to analyze it on my machine

(In reply to comment #0)
> Description of problem:
> vm is hang after migration on 5.4 host
> 
> Version-Release number of selected component (if applicable):
> kvm-83-60.el5
> 
> How reproducible:
> always
> 
> Steps to Reproduce:
> 1. command on source host:
> # qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack -m 1G -uuid `uuidgen` -net
> nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net
> tap,vlan=0,script=/etc/qemu-ifup  -drive file=rhel-5.3-server-64.qcow2,if=ide
> -boot c -vnc :10
          
               ^^^^

> 2. command on destination host:
> # qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack -m 1G -uuid `uuidgen` -net
> nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net
> tap,vlan=0,script=/etc/qemu-ifup  -drive file=rhel-5.3-server-64.qcow2,if=ide
> -boot c -incoming tco:0:4567 -vnc :10

                       ^^^          ^^^^^
> 3.

I can't see how this works since according to this cmd line you use the same host for the destination and both src/dst use -vnc :10.

(In reply to comment #13)
> (In reply to comment #0)
> > Description of problem:
> > vm is hang after migration on 5.4 host
> > 
> > Version-Release number of selected component (if applicable):
> > kvm-83-60.el5
> > 
> > How reproducible:
> > always
> > 
> > Steps to Reproduce:
> > 1. command on source host:
> > # qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack -m 1G -uuid `uuidgen` -net
> > nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net
> > tap,vlan=0,script=/etc/qemu-ifup  -drive file=rhel-5.3-server-64.qcow2,if=ide
> > -boot c -vnc :10
> 
>                ^^^^
> 
> > 2. command on destination host:
> > # qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack -m 1G -uuid `uuidgen` -net
> > nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net
> > tap,vlan=0,script=/etc/qemu-ifup  -drive file=rhel-5.3-server-64.qcow2,if=ide
> > -boot c -incoming tcp:0:4567 -vnc :10
> 
>                        ^^^          ^^^^^
> > 3.
> 
> I can't see how this works since according to this cmd line you use the same
> host for the destination and both src/dst use -vnc :10.  

sure at the difference machines. 
destination host just listen for source host. so use tcp:0:4567, and at source host (qemu) migration -d tcp:destinacion_ip:4567

there are at different machine so can use -vnc :10 at the same time
there

host: AMD 
=>pass

host: intel (not i7)
=>pass

host: intel (i7):
ept enabled -> ept disabled => failed
ept disabled -> ept disabled => failed
ept enabled -> ept enabled => failed
guest include rhel5.3, rhel5.4,rhel4.8, window2008 => all failed

(In reply to comment #15)
> host: AMD 
> =>pass
> 
> host: intel (not i7)
> =>pass
> 
> host: intel (i7):
> ept enabled -> ept disabled => failed
> ept disabled -> ept disabled => failed
> ept enabled -> ept enabled => failed
> guest include rhel5.3, rhel5.4,rhel4.8, window2008 => all failed  

Did you use -cpu qemu64,sse2 ? I don't see it above.
Please retest with that.

1. command used:
source host:
qemu-kvm -no-hpet -cpu qemu64,+sse2 -rtc-td-hack -m 1G -uuid `uuidgen` -net nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net tap,vlan=0,script=/etc/qemu-ifup -drive file=RHEL-5.4-32.qcow2,if=ide,boot=on -boot c -vnc :6

destination host:
qemu-kvm -no-hpet -cpu qemu64,+sse2 -rtc-td-hack -m 1G -uuid `uuidgen` -net nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net tap,vlan=0,script=/etc/qemu-ifup -drive file=RHEL-5.4-32.qcow2,if=ide,boot=on -boot c -vnc :8 -incoming tcp:0:4586

2. Test on Beta5:
(qemu)info migration
Migration status: failed

3. Test on 5.4 nightly(0602)
after migration, destination guest is black, source guest hang, monitor can not be operated on source guest


you can try on my machine
10.66.70.6 & 10.66.70.31 
passwd: redhat

command used:
source host:
#qemu-kvm -no-hpet -cpu qemu64,+sse2 -rtc-td-hack -m 1G -uuid 2406a5a1-d905-4f42-bf11-b1f6069fa8d3 -net
nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net
tap,vlan=0,script=/etc/qemu-ifup -drive file=RHEL-5.4-32.qcow2,if=ide,boot=on
-boot c -vnc :6

destination host:
#qemu-kvm -no-hpet -cpu qemu64,+sse2 -rtc-td-hack -m 1G -uuid 2406a5a1-d905-4f42-bf11-b1f6069fa8d3 -net
nic,model=e1000,macaddr=00:1a:4a:16:97:86,vlan=0 -net
tap,vlan=0,script=/etc/qemu-ifup -drive file=RHEL-5.4-32.qcow2,if=ide,boot=on
-boot c -vnc :8 -incoming tcp:0:4568

I just run it now, hope we didn't kill the image since the guest cannot even boot. The kernel crashed.
Can you re-start with a fresh image?
Also, DO NOT use boot=on for ide.

Can not reproduce the issue on a i7 5U4 host. migaration is successfully done.
cpu usage change from 0.3%~0.9% to 0% on src host,and from 0% to 0.3% ~1% on dst host.

Tested ping-pong migration
1. enable_ept=1 <==> enable_ept=1
2. enable_ept=1 <==> enable_ept=0
3. enable_ept=0 <==> enable_ept=0

[root@intel-i7-12-2 ~]# dmesg | tail
switch: port 2(vm1) entering learning state
vm1: no IPv6 routers present
switch: topology change detected, propagating
switch: port 2(vm1) entering forwarding state
kvm: 22176: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 22176: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffd76970
kvm: 22176: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079
kvm: 22176: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 22176: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0xffd76970
kvm: 22176: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x530079


CLI 1 . a fresh install 5u3 guest using ide block and e1000 NIC
[root@intel-i7-12-2 images]# /usr/libexec/qemu-kvm -cpu qemu64,+sse2 -drive file=rhel5u3-server-x86_64.qcow2,cache=off,format=qcow2,index=0 -no-hpet -usbdevice tablet -rtc-td-hack -name vm1 -smp 2 -m 2048 -boot c -net nic,vlan=1,macaddr=00:1a:4a:fe:32:01,model=e1000 -net tap,vlan=1,ifname=vm1,script=/etc/qemu-ifup -vnc :12 -monitor stdio -incoming tcp:0:5800

CLI 2 . a fresh instll 5u4server guest using virtio block and virtio NIC
[root@intel-i7-12-2 images]# /usr/libexec/qemu-kvm -cpu qemu64,+sse2 -drive file=rhel5u3.virtio,if=virtio,boot=on,cache=off,format=qcow2,index=0 -no-hpet -usbdevice tablet -rtc-td-hack -name vm1 -smp 2 -m 2048 -boot c -net nic,vlan=1,macaddr=00:1a:4a:fe:32:02,model=virtio -net tap,vlan=1,ifname=vm1,script=/etc/qemu-ifup -vnc :12 -monitor stdio -incoming tcp:0:5800


qemu monitor cmd: migrate  -d tcp:$dst_ip:5800


[root@intel-i7-12-2 images]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.4 Beta (Tikanga)
[root@intel-i7-12-2 images]# rpm -q kernel kvm
kernel-2.6.18-152.el5
kvm-83-77.el5

the same testing is running on rhevh 5.4-2.0.99 (7.1). result coming soon.

> 
> CLI 1 . a fresh install 5u3 guest using ide block and e1000 NIC
> [root@intel-i7-12-2 images]# /usr/libexec/qemu-kvm -cpu qemu64,+sse2 -drive
> file=rhel5u3-server-x86_64.qcow2,cache=off,format=qcow2,index=0 -no-hpet
> -usbdevice tablet -rtc-td-hack -name vm1 -smp 2 -m 2048 -boot c -net
> nic,vlan=1,macaddr=00:1a:4a:fe:32:01,model=e1000 -net
> tap,vlan=1,ifname=vm1,script=/etc/qemu-ifup -vnc :12 -monitor stdio -incoming
> tcp:0:5800
> 
> CLI 2 . a fresh instll 5u4server guest using virtio block and virtio NIC
> [root@intel-i7-12-2 images]# /usr/libexec/qemu-kvm -cpu qemu64,+sse2 -drive
> file=rhel5u3.virtio,if=virtio,boot=on,cache=off,format=qcow2,index=0 -no-hpet
> -usbdevice tablet -rtc-td-hack -name vm1 -smp 2 -m 2048 -boot c -net
> nic,vlan=1,macaddr=00:1a:4a:fe:32:02,model=virtio -net
> tap,vlan=1,ifname=vm1,script=/etc/qemu-ifup -vnc :12 -monitor stdio -incoming
> tcp:0:5800
> 

also tested 4G/8G mem. => pass 
(host has 12G RAM)

Can not reproduce this issue on non Intel i7 host.
and According comment #15 , update the bug summary.


CUP info: 

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
stepping	: 4
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5319.89
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management: [8]

when I running the testing on :

[root@dhcp-66-70-6 ~]# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 5.4-2.0.99 (7.1)
[root@dhcp-66-70-6 ~]# rpm -q kvm
kvm-83-77.el5
[root@dhcp-66-70-6 ~]# rpm -q kernel
kernel-2.6.18-154.el5

I also can not reproduce the original bug ( cpu usage reach 100%) 

CLI:
root     30111 29041  0 13:35 pts/3    00:00:01 /usr/libexec/qemu-kvm -drive file=/mnt/rhel53-64-ser-virtio.qcow2,if=virtio,cache=off,index=0,boot=on -net nic,macaddr=20:20:20:00:18:59,model=virtio -net tap,script=/etc/qemu-ifup -rtc-td-hack -no-hpet -usbdevice tablet -cpu qemu64,+sse2 -smp 2 -m 4096 -vnc :12 -monitor stdio -boot c -incoming tcp:0:5800


but there is another non-100% reproducible issue. 
opened a new bug for that issue :

  Bug 507659 -  Migrate command not end and vm responseless on Nahalem host

So can you close this bug?

Please reopen if you can reproduce the original issue